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Executive Summary 



This Executive Summary forms part of the final report of the findings of the independent 
external evaluation of the NRICH online mathematics project. The evaluation was 
commissioned by the University of Cambridge, UK as represented by the Millennium 
Mathematics Project. The evaluation was carried out by the evaluation team from the 
University of Southampton, UK, during 1998-99. Data collection took place between 
April and September 1999. An interim report, based on one element of the evaluation, 
was presented at the NRICH conference held in July 1999. 

1 . The NRICH online mathematics enrichment project began in 1996 with the aim of 
establishing a permanent national UK centre for curriculum enrichment in 
mathematics. The project aims to provide mathematical learning support for very able 
children of all ages through the publication of a regular web-based ‘magazines’ 
featuring mathematical problems, puzzles, articles and games. University mathematics 
students act as peer teachers providing an electronic answering service. The centre 
also offers support, advice and inservice training to teachers, and resources for 
mathematics clubs. The NRICH website can be found at http://nrich.maths.org.uk/ 

2. Internal evaluations of the project, earned out in 1997 and 1998, suggested that an 
increasing number of people were accessing the NRICH project website. The 
evaluations also indicated that teachers using the NRICH materials were generally 
satisfied with the type of problems presented, and that pupils who worked on the 
problems developed a richer view of mathematics. 

3. The objectives of the external evaluation were to assess how the use of the NRICH 
website facilities enhances the mathematical development of children who have the 
potential to go on to study mathematical subjects at university, how the features of 
the website are used by teachers to help meet the special educational needs of 
exceptionally able children in mathematics, and the particular contribution of 
Information Communications Technology to the above. These objectives were 
derived from the aims of the NRICH project. 

4. The evaluation design incorporated a range of methods to provide data on the 
evaluation objectives. The various elements of the evaluation were an analysis of the 
responses to questionnaires completed by pupils, teachers, and other interested parties 
(such as parents) who access the NRICH website, a critical review of the NRICH 
website, together with selected case studies of school, classroom and pupil use of the 
NRICH project facilities. 

5. Analysis of questionnaires completed by 199 pupils, 450 teachers, and 67 ‘friends of 
NRICH’ (such as parents) revealed that most NRICH users lived in England. This 
was particularly true of teachers. Over two-thirds of the pupils using NRICH were 
boys. Approximately the same proportion was white. A large proportion of users 
accessed NRICH at home. The majority of teachers worked in the state sector (both 
primary and secondary); ten percent were from private schools. The majority of users 
of all types were relatively new to NRICH, having been accessing the website for six 
months or less. The most frequent reason for accessing the NRICH website was to 
use it as a source of interesting mathematical problems. Most users were not 
registered with NRICH (a no-fee option open to all). The NRICH site was 
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complimented by all categories of user as providing interesting problems, being 
attractively presented and generally easy to navigate. 

6. The NRICH website was judged by the evaluation team to score highly on each of the 
website evaluation criteria. The new design of the site, launched in July 1999, was 
judged to be attractive, functional, easy to navigate, and contain high-quality 
materials. NRICH compared very favourably with other sites that provide 
mathematical puzzles, games and problems, and/or an answering service. The NRICH 
server statistics showed an increase in accesses to the site which was likely to be the 
result of more people accessing the NRICH site more often. 

7. In each of the three case study schools, at least one teacher made regular and often 
frequent use of the NRICH website, though none made use of the wider NRICH 
facilities available to registered teachers. NRICH was mainly used a source of 
interesting mathematical problems. Pupil usage of NRICH in the schools was much 
more varied. Only a very few pupils were aware of NRICH and had accessed the site 
themselves. While there was some evidence of impact on more able pupils, none of 
the teachers were able to quantify this impact but all praised NRICH as a very 
valuable resource. 

8. The three selected case profiles of pupil usage of NRICH revealed that these 
particular pupils accessed NRICH no more than once a month. All found the ‘one-to- 
one’ facility, where they could pose questions to University students, helpful and 
informative. Such exchanges often left the pupils wanting to know more, a situation 
the pupils viewed as positive. While few of the exchanges were related to the 
mathematical problems provided on the NRICH site, all these pupils valued the 
opportunity of being able to ask questions and receive replies. 

9. The main impact of NRICH on the more able pupils was in terms of helping them to 
gain a wider appreciation of mathematics and raising the profile of mathematics as a 
subject that could be interesting enough to pursue either within or outside school or 
for further study. Quantifying this impact was beyond the scope of this evaluation. 
Teachers mostly accessed NRICH to find problems to use in their teaching. The 
teachers used a variety of approaches to meet the needs of their more able pupils. 
Some used the NRICH problems with groups of more able children withdrawn from 
their regular classrooms. Some teachers used NRICH problems as extension material 
once regular classwork was complete. For some, NRICH was one resource amongst 
many. Only a few organised an extra-curricular mathematics club based solely around 
NRICH. The contribution of information communications technology (ICT) to both 
the enhancement of pupils’ mathematical development and to how teachers made use 
of the NRICH facilities was associated with the functionality and accessibility of the 
NRICH site. The interaction that was possible through using ICT was seen as a 
particular advantage of the NRICH project. 

Keith Jones and Helen Simons, University of Southampton, UK 
October 1999 

The full report is available from: The Research and Graduate School of Education, 
University of Southampton, Highfield, Southampton, S017 1BJ, UK. 

Telephone: +44 (0)23 80 593475 Fax: +44 (0)23 80 593556 
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1. Introduction 

This report presents the findings of the independent external evaluation of the NRICH 
online mathematics project. The evaluation was commissioned by the University of 
Cambridge, UK as represented by the Millennium Mathematics Project. The evaluation 
was carried out by the evaluation team from the University of Southampton, UK, during 
1998-99. Data collection took place between April and September 1999. An interim 
report, based on one element of the evaluation, was presented at the NRICH conference 
held in July 1999. 

1 . 1 Meeting the Needs of the More Able Learner 

The particular needs of the more able learner across the school curriculum have been an 
issue of concern for some time and have been recently highlighted in the UK by the 
appointment by the Department for Education and Employment of a new advisory group 
on gifted and talented children (DfEE press release 413/98, September 4th 1998), and 
the publication of the House of Commons Education and Employment Committee report 
Highly Able Children (House of Commons Education and Employment Committee 
1999) and an Ofsted-funded research review, Educating the Very Able, (Freeman 1998). 
Most recently, the DfEE initiative ‘Excellence in Cities’ has included a “gifted and 
talented children strand” aimed at supporting inner-city schools to make effective 
provision for their most able children (DfEE 1999). As part of this initiative there is a 
requirement that every school has a clear policy to ensure that its most able pupils are 
fully stretched to achieve their potential. Funding is available for mentoring programmes 
and for ‘Gifted And Talented Summer Schools’, which are being piloted in 1999. ‘World 
class tests’, designed to challenge the most able, initially in mathematics and problem- 
solving, are also under development by the UK Qualifications and Curriculum Authority. 

As Freeman (1998 p44) describes, enrichment “is the deliberate rounding out of the basic 
curriculum subjects with ideas and knowledge that enable a pupil to be aware of the 
wider context of a subject area”. It is an approach that is consistently advocated for the 
more able (see, for example, Eyre and Maijoram 1990, Koshy and Casey 1997, Shore 
1991). Enrichment has also been recommended in various ways in the teaching and 
learning of mathematics over a number of years (see, for example, House 1987, Kennard 
1996, Sheffield 1999, Straker 1983). 

1.2 Online Mathematics Enrichment 

The central aims of the NRICH online mathematics enrichment project are to “promote 
an interest in mathematics, to raise the standards of achievement in school mathematics, 
to assist the mathematical development of children who have the potential to go on to 
study mathematical subjects at university, and to support the special educational needs of 
exceptionally able children” (NRICH statement of aims). The principle method of 
meeting these aims is through the provision of online mathematics ‘magazines’ enhanced 
by university students acting as peer-teachers providing an electronic answering service 
for learners. The web address 1 for the NRICH website is http:Wnrich.maths.org.uk. Work 
on the project began in 1996 and the project was internally evaluated in 1997 and 1998 
(Jared 1997, 1998). The development phase of the NRICH project ended in August 



1 or URL, unif orm (previously ‘universal’) resource locator. 
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1999. A second phase began in September 1999, under the umbrella of the newly- 
established Millennium Mathematics Project 2 . 

The establishment of an online mathematics enrichment project is a timely development 
for two reasons. First, there is increasing availability and sophistication of information 
and communication technology (ICT). In the UK, the use of such technology is being 
encouraged by central government initiatives such as the National Grid for Learning 
(DfEE 1997) and the use of the New Opportunities Fund to provide ICT training for 
teachers and school librarians. Second, there exist a range of concerns regarding, for 
instance, the overall quality of the teaching and learning of mathematics, the specification 
of the mathematics curriculum, the uptake of mathematics post-16, and provision for the 
more able pupil in mathematics (see, for example, London Mathematical Society et al 
1995). Many of these concerns are not limited to the UK, but, following the results of 
the Third International Mathematics and Science Study, are present in many other 
countries (see, for example, Schmidt et al 1999). 

The NRICH online mathematics project is an attempt to harness the ‘information 
highway’ both through the publication of web-based magazines published monthly and 
the provision of e-mail discussion facilities and a pupil answering service. More details of 
the project, including its mission statement and its aims and objectives are provided in 
Appendix A of this report. The previous internal evaluations (Jared 1997, 1998) have 
provided valuable information on the progress in establishing the project. The 1998 
evaluation showed that ‘hits’ on the NRICH website reached around 24000 a month 
from February 1998 onwards (although website statistics in the form of ‘hits’ need to be 
treated with caution as they do not say anything about user impact). Another indication 
of the impact so far of the NRICH project is that, by May 1998, 408 teachers and 25 1 
pupils had registered with NRICH, a no-fee option giving them access to the NRICH e- 
mail facilities. A questionnaire, available on the NRICH website during May and June 
1998, was completed by 48 teachers and this indicated satisfaction with the 
appropriateness of the mathematics problems that appear in the monthly magazine. Data 
from a similar questionnaire completed by 68 pupils suggested that NRICH was being 
successful in enhancing the pupil view of mathematics by showing that mathematics can 
be fim and that mathematical problems can be approached in a variety of ways. 

These earlier internal evaluations indicated that the NRICH project might profitably 
consider how best to convey the NRICH objectives to its intended audience, and give 
further thought to the presentation of some of the mathematical problems and solutions 
and the promotion and management of the e-mail mailing lists. 



2. Aims and Objectives of the 1998-99 External Evaluation 

The aim of the independent external evaluation undertaken during 1998-9 was to 
examine what the NRICH project had achieved to date in order to inform the future 
development of the project. In particular, the intention was to audit the success of the 
project in meeting some of its stated aims. The evaluation also sought to document any 



2 For details of the Millennium Mathematics Project see http://mmp.maths.org/ 
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unanticipated outcomes of the project. The aims of the NRICH project are reproduced in 
appendix A of this report. 

The evaluation focused on the success of the NRICH online mathematics project in the 
following areas (all taken from the aims and objectives of the NRICH project): 

• assisting the mathematical development of children who have the potential to 
go on to study mathematical subjects at university, 

• supporting the special educational needs of exceptionally able children, 

• developing the use of Information Communications Technology to provide 
interactive links to the NRICH Centre and to facilitate links between schools 
and between individual children. 

Correspondingly, the objectives of the evaluation of the NRICH project were to assess. 

1 . how the use of the NRICH website enhances the mathematical development of 
children who have the potential to go on to study mathematical subjects at 
university 

2. how the features of the website are used by teachers to help meet the special 
educational needs of exceptionally able children in mathematics 

3. the particular contribution of Information Communications Technology to the 
above 

Given the inter-related form of objectives, the evaluation was planned as a number of 
complementary elements, outlined in the following section. 

3. Evaluation Design and Methodology 

3. 1 Research on Teacher and Pupil Usage of ICT 

The design of the evaluation was informed by previous research on teacher and pupil 
usage of ICT. Data from surveys on the use of information and communication 
technology (ICT) in schools show that, despite continuing investment in resources, the 
percentage of headteachers reporting that ICT is making a significant contribution to 
teaching in their institutions has fallen in the period 1988-98 (Lynch 1999). In secondary 
schools it is down from a peak of 40% in 1990 to nearer 20% in 1998. In primary 
schools it is down from a peak of about 30% in 1992 to nearer 15% in 1998. One reason 
suggested for this is that, through rapid technological development, a significant 
proportion of teachers indicate that they feel they lack the appropriate capability in their 
ICT use. For example a recent Technology Colleges Trust survey found in 1997 that 
“fewer that one-fifth of teachers had sufficient confidence and competence in the use of 
generic IT applications to enable them to apply applications or to develop IT capability in 
pupils” (quoted in Lynch 1999). 

Another reason may relate to teachers’ access to appropriate levels of resource. For 
example, in terms of access to the internet, Jervis and Steeg .(1998) found that while 83% 
of secondary schools had some form of internet access, most had a connection for one or 
two stand-alone computers, often in ‘staff-only’ areas. The most frequent reason for 
limiting pupil access was logistic: a lack of machines with internet access. While the 
recent DfEE survey also showed that 83% of secondary schools have internet access, for 
primary schools the comparative figure is 1 7% (DfEE 1 998). On the other hand there is 
evidence that access is rapidly developing. According to the British Educational 
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Communications Technology Agency (BECTa), in a period of two years, secondary 
school access has doubled and primary school access has grown six fold, while the 
National Grid for Learning website is getting over a million ‘hits’ per week (Lynch 
1999). By October 1999, the Times Educational Supplement was reporting that 93% of 
secondary schools were connected to the internet, with the corresponding figure in 
primary schools being 62% (TES, 15 October 1999). 

Despite these headline figures, a further reason for the lack of impact of ICT to date may 
relate to the type of equipment generally available in schools. For example, the recent 
Research Machines survey found that although Britain leads the world in the provision of 
ICT in schools, being the only G7 nation where all schools have a least one computer 
and with the highest percentage of secondary schools with one or more multimedia 
machines, almost 40 per cent of British school computers are at least 5 years old 
(Research Machines 1999). Such machines are unlikely to have internet capability. Thus, 
while an increasing number of schools are becoming connected to the ‘information 
highway’, for individual teachers there may still be considerable practical barriers to their 
effective use of internet-based resources. 

3.2 Design Rationale 

Given that teacher competency, levels of resources, and internet access are likely to 
improve over the short to medium term (the first through the UK ‘New Opportunities 
Fund’ provided ICT training for teachers, the latter two through UK government- 
supported resource and infrastructure provision), the external evaluation of the NRICH 
project focused on the regular users of the NRICH site, ones for whom competency and 
access were not likely to be major issues. Regular users were also judged to be likely to 
be registered with NRICH and thereby users of the wider NRICH facilities (such as the 
e-mail bulletin boards and the answering service), A wider focus for the evaluation would 
have spread the evaluation resource too thinly and, in so doing, have revealed only what 
is already known, that competency and access are the major limiting factors. Focusing on 
regular users allowed for richer data to be gathered on their usage which should be of 
more use in informing future development of the NRICH project. At the same time, 
analysis of regular users allowed some informed inferences to be made about who is not 
accessing the site, which is also likely to be of use in informing the further development 
of the project. 

Taking into account the evaluation objectives, and the requirements of validity and 
reliability, the design of the evaluation needed to encompass a range of methods. Where 
appropriate, the research techniques exploited the functionality of the internet as, by 
definition, regular NRICH users are internet users. For example, the world wide web is 
increasingly being used for survey research. Bertot and McClure (1996) point to both the 
advantages and disadvantages of web-based questionnaires while the research of 
Buchanan and Smith (1999) suggests the possible superiority of web-generated data. 
Schmidt (1997) outlines the steps that can be taken to minimise any potential problems 
with generating data in this way. Similar issues arise in using electronic mail for 
interviewing (Roselle and Neufeld 1998). 

In planning the overall design of the evaluation, due attention was paid to ethical 
considerations. A conscious decision was made to avoid using any personal information 
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on individuals that those individuals, purposefully or not, had provided to NRICH 
through registering with NRICH or communicating electronically with the NRICH 
website. The precise design of the evaluation, and more detail on the validation of the 
research and on the ethical considerations which informed the design, implementation 
and reporting of the research, are covered in detail in the following sections. 

3.3 Evaluation Design 

The evaluation design incorporated a range of methods to provide data on the evaluation 
objectives. The four elements of the evaluation were website impact, website evaluation, 
short case studies, and case profiles. Each of these elements is described and justified 
below. 



3.3.1 Element 1: website impact 

In order to gauge the usage of NRICH, the major component of the evaluation was a set 
of three questionnaires mounted on the NRICH site to coincide with the publication of a 
new edition of the online magazine. Three versions of the questionnaire were developed, 
one for school pupils or students to complete, one for school or college teachers, and 
one for others who accessed the NRICH site (parents, local authority advisors, higher 
education staff, etc). The questionnaires were mounted on the NRICH website during 
May 1999 (from 1 May 1999 to 6 June 1999). 

There is much to recommend web-based questionnaires, particularly when the target 
population is well-defined (Schmidt 1997), as NRICH users are. Amongst the benefits 
are access to a large sample of individuals, depending on how the questionnaire is 
administered. For this research project, and with the assistance of the NRICH technical 
team, the NRICH website was arranged so that anyone accessing the NRICH site at any 
point during the designated period would be presented with the questionnaire at the time 
of access in the form of a ‘pop-up window’. The intention of this admittedly intrusive 
technique was to maximise the number of completed questionnaires and hence generate a 
more representative set of data. NRICH users were forewarned by a notice mounted on 
the website during the previous month and asked for their cooperation. 

A second benefit of web-based questionnaires is that data from the questionnaires is 
logged as the questionnaires are completed thus removing any problems associated with 
data entry (for instance, transcription errors). A third benefit, according to the research 
of Buchanan and Smith (1999), is obtaining increased levels of honesty and self- 
revelation through using computer-based questionnaires. Despite these advantages there 
are well-documented potential problems (Buchanan and Smith 1999, Schmidt 1997). 
These include incomplete responses, unacceptable or frivolous responses, and multiple 
submissions from the same respondent. 

Such potential problems mean that close attention was paid to the data set generated by 
the questionnaires. It was anticipated that the use of the ‘pop-up window’, which 
appeared whenever someone accessed the NRICH website , was likely to increase the 
number of incomplete responses. The use of a ‘cookie’, a small data file sent by the 
NRICH server to any computer which accesses the NRICH site, was considered. This 
would allow the NRICH server to distinguish between computers that had accessed the 
NRICH site, and hence been presented with the ‘pop-up’ questionnaire, and those that 
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had not. The ‘cookie’ could be set so that only the first access through a particular 
computer would get the ‘pop-up’ questionnaire. The advantage of such a mechanism is 
that the ‘pop-up’ questionnaire does not appear every time the NRICH site is accessed 
by a user using a given computer. However, the use of a ‘cookie in this way only 
distinguishes between computers and not between users. It would mean that only the first 
user gets the questionnaire; other users of the same computer would not. As it was likely 
that school use of NRICH entailed several users using the same computer at various 
times, and because, for the home user and for some schools, many ‘internet service 
providers’ route their web traffic through a ‘webcache’ or ‘proxy’ servers 3 , which would 
appear to the NRICH server as just one ‘user’, the use of a ‘cookie’ would likely 
severely intrude on the number of users who were presented with the questionnaire. In 
addition, it is known that many users set their web browser to reject the downloading of 
cookies (through some concern about the intrusive nature of cookies, plus possible 
security and virus issues) ( Stout 1997, p84-86). Since NRICH has not, this far, made 
much use of ‘cookies, such a use for the questionnaires may have deterred some users 
from using the NRICH website again because these users had concerns about the use of 
‘cookies’. This was not an acceptable outcome. The decision of the evaluation team was 
to reject the use of a ‘cookie’ and accept a potentially high number of incomplete 
responses as regular users, who were repeatedly presented with the questionnaire, or 
internet browsers who just happened on the site, clicked on ‘questionnaire complete’ as a 
way of getting rid of the ‘pop-up window’. 

The questionnaires were constructed to provide data on the usage and impact of NRICH. 
Standard techniques were adopted to develop and test the questionnaires (Oppenheim 
1992, Dillman 1999), with piloting of both paper versions and electronic versions in 
advance of formally mounting them on the NRICH website. As a result of the piloting, 
some modification was made to the form and length of the questionnaires. In particular, 
questions relating to the socio-economic class of the pupils were omitted to shorten the 
pupil questionnaire. Similarly, some questions were compacted together to reduce the 
number of different sorts of questions even though this could affect the clarity of a few of 
the individual questions. NRICH users were advised some time in advance as to the 
purpose of the questionnaires and how they would appear on the NRICH site. 
Confidentiality of the response data was guaranteed. Through these methods the ethical 
issues associated with such data collection techniques were addressed. 

The questionnaires covered type of access, form of usage of NRICH material, and 
evaluative comment on the material. The questionnaires also sought information on the 
type of school and household (or other location) where NRICH material was accessed. 
Respondents could also provide their NRICH ID (if known), as a way of ascertaining 
what proportion of the registered users completed the questionnaire, and offer to be 
contacted again by e-mail to provide follow-up information. Again, respondents were 
assured of the confidentiality of this data. The results of the analysis of the questionnaires 
are presented in section 4 of this report. The questionnaires are provided in Appendix B. 



3 a technique involving holding a copy of recently accessed data, in a way designed to speed up 
subsequent access to the same data by anyone routed through that server. 
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3.3.2 Element 2: website evaluation 

With internet-based resources developing apace, there now exist a range of websites that 
provide mathematics problems, solutions, and assistance. Thus an important element of 
the design focused on comparing the NRICH provision with other internet-based 
mathematics problem-solving sites. A range of website evaluation procedures are 
available and suitable criteria were adapted from those described by Branch et al (1999), 
Coe and Land (1998), Shneiderman (1997), and by Testa (1998). In addition, this part of 
the evaluation included an analysis of the frequency and nature of the accesses to the 
NRICH website, accompanied by trends over time, using techniques described by Bertot 
et al (1997), Buchanan and Lukaszewski (1997) and Stout (1997). 

The world wide web has been variously described as a ‘junkyard’ or ‘like a library where 
everything has been thrown on the floor’. In the absence of universally recognised web 
standards, evaluative criteria for Web sites have been developed and these generally 
include the following factors: 

• Authority 

• Accuracy 

• Currency 

• Navigation and Design 

• Applicability and Content 

• Scope 

• Audience Level 

• Quality of Writing 

Associated with each of these factors is a list of questions that serve to focus the 
evaluative judgement. The full evaluative criteria are given in Appendix C. The results of 
applying these criteria to the NRICH site are presented in section 5. 

An analysis of the frequency and nature of the accesses to the NRICH website, 
accompanied by trends over time, is possible because Web servers, in general, provide a 
log (or logs) in which site accesses and events are stored. The format of these records 
can vary, depending on the server, but generally include: 

• date and time records, making it possible to determine the most common access 
times and days 

• visitor domains, indicating where visitors are accessing the site from 

• the files accessed or downloaded 

• errors and alerts, providing a measure of site reliability 

It has to be stressed, however, that such log files are far from straightforward to interpret 
in their raw form (Bertot et al 1997, Buchanan and Lukaszewski 1997, Stout 1997). It is 
particularly important, for example, to try to distinguish between ‘hits’ (downloads of 
single item files from an HTML page - of which there can be many 4 ) and ‘accesses’ 
(downloads of complete web pages). As a consequence of this, and because many 
internet providers route their internet traffic through ‘proxy’ servers 5 , the logs from any 
particular website do not easily or accurately trace the number of users, but rather they 
reflect the number of accesses by the referring server. More importantly, however, such 



4 each image on a page can count an a separate file, for example. 

5 a technique involving holding a copy of recently accessed data, in a way designed to speed up 
subsequent access to the same data by anyone routed through that server. 



Online Mathematics Enrichment 



logs do not say anything about what was done with the information obtained from the 
website by the person accessing the site. Indeed, the person may have hardly glanced at 
the content, never mind made any use of it. Data from the log files of the NRICH server 
are presented and interpreted in section 5 of this report. 



3.3.3 Element 3: short case studies 

The intention of this element of the evaluation was to provide evidence of the impact of 
NRICH on the quality of teaching and learning in the classroom and to ascertain how 
schools supported the more able pupil in mathematics. This was achieved primarily 
through interviewing a small, random sample of teachers and pupils in schools and 
observing classes at work. The focus was on teachers in England who made up over 
75% of respondents to the questionnaire (see Section 4.2). As a first step, a stratified list 
of teachers, who had been accessing NRICH for six months or more, was prepared from 
those who, when completing the teacher questionnaire, had offered to provide further 
information (a total of 63 teachers from England out of the 343 who had completed the 
questionnaire). Only 14 of these teachers said that they were registered with NRICH, a 
no-fee option giving them access to the NRICH e-mail facilities. The stratification 
employed was primary, secondary, and private school. Details of the numbers-of teachers 
contacted and the results of this contact are given in Table 1. 



Table 1: Number of teachers contacted for school visits 





Teachers 
offering to 
be contacted 


Incorrect 

e-mail 

addresses 


Nil 

responses 


Offers of 
more 

information 
but not visits 


Offers 
to visit 
the 

school 


Schools 

visits 

made 


Primary 

schools 


25 


2 


10 


9 


4 


2 


Secondary 

schools 


28 


2 


18 


3 


5 


2 


Private 

schools 


10 


0 


9 


0 


1 


1 



A total of 25 teachers from middle, junior or infant schools were identified as having 
been accessing NRICH for six months or more. These 25 teachers were contacted by e- 
mail and asked if a visit to their school might be arranged. Four responded positively. 
From ten there was no response to the e-mail and two addresses provided were 
incorrect. The remaining were happy to provide further information by e-mail but said, 
for various reasons, that a school visit would not be possible. 

Twenty-eight secondary school teachers, who had been accessing NRICH for six months 
or more, indicated, at the time of completing the teacher questionnaire, that they were 
willing to provide further information. These 28 teachers were contacted by e-mail and 
asked if a visit to their school might be arranged. Five responded positively. From 
eighteen there was no response to the e-mail and two addresses provided were incorrect. 
As with the primary teachers, the remaining were happy to provide further information 
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by e-mail but said, for various reasons, that a school visit was not convenient. The 
corresponding figures from private schools were that 10 offered to be contacted. When 
e-mailed only one responded positively to arranging a school visit. One address was 
invalid; no response was obtained from the other teachers. Teachers from 16-19 or 
further education colleges and from Special schools who indicated that they were happy 
to be contacted by e-mail were not included in this element of the evaluation. 

A total of five schools were visited, two primary, two secondary, and one private school. 
One school was visited twice. Data, in the form of audio-taped interviews and associated 
notes with teachers and pupils where possible, classroom observation notes where 
possible, and the records of e-mail interviews, were collected. Interview and observation 
schedules are given in Appendix D. It was not possible to collect the same amount and 
quality of information in each school. Consequently, three short case studies are 
presented in section 6; one of a primary school, one of a secondary school, and one of a 
private school. A further short case study focusing on a secondary 11-18 school that uses 
both PASSMaths 6 and NRICH will become available when the evaluation of the 
PASSMaths project, being carried out by the evaluation team at the time of writing, is 
completed and published. 



3.3.4 Element 4: case profiles 

Element 4 of the evaluation was planned to be in the form of reasonably indepth 
accounts of the experiences of a small sample of pupils in accessing the NRICH website, 
particularly their experience in using the ‘one-to-one’ facility where they can talk over a 
mathematical problem with someone studying mathematics at university (one of the 
distinctive features of the NRICH website). The intention was to attempt to provide an 
element of longitudinal data to inform evaluation objectives 1 and 3. 

In order to use this ‘one-to-one’ facility, and ‘NRICHtalk’, where pupils can engage in 
discussion with other pupils, pupils have to register with NRICH, a straightforward 
process with no cost involved. The pupil questionnaire asked pupils to indicate if they 
were registered and, further, if they were happy to provide further information. A total of 
199 pupil responses to the pupil questionnaire were accepted for analysis (see section 4.1 
for further details) but of these only twelve of the pupils who said that they were 
prepared to be contacted for further information said that they were registered with 
NRICH (see Table 2 below). 



Table 2: Number of pupils contacted for further information 





Registered with NRICH 




Happy to be e-mailed 


not specified 


yes 


Totals 


not specified 


121 


8 


129 


yes 


58 


12 


70 


Totals 


179 


20 


199 



6 PASSMaths is an online mathematics magazine aimed at a slightly older audience than NRICH. As of 
September 1999 it, like NRICH, is a component of the Millennium Mathematics Project 
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Of these twelve pupils who said that they were registered with NRICH, seven were from 
the UK and these were contacted by e-mail first. They were asked if they would be 
prepared to share with the evaluation team their experience of using NRICH. Two made 
an initial reply in the affirmative but then only one reply was received from one of these 
pupils to follow-up messages. No reply to the first message was received from any of the 
other five pupils. 

Consequently, the selection was widened to include all 70 pupils who had provided an e- 
mail address in the hope that some of them might be registered with NRICH even though 
they did not state this on the questionnaire. From these 70 messages, four replies were 
received; from the remaining no reply was received (three e-mail addresses were invalid). 
Of the four pupils who replied, two were registered with NRICH and had used ‘one-to- 
one’ and ‘NRICHtalk’, although one of them not for almost a year and not now as the 
person had left school. The other two replies were from pupils who were not registered 
and had never used any of the NRICH bulletin boards. 

Through these efforts, some data was gathered on the use made of NRICH by three 
pupils, one a home user of primary age, one a secondary school pupil who accessed 
NRICH from the school IT suite, and one a student at a 16-19 college who accessed 
NRICH at home. Unfortunately, the available data is not rich enough to provide 
individual case profiles, but some account of the experiences of the NRICH user is 
provided in section 7 of this report. 



3.4 Validation Procedures 

The validity and reliability of the evaluation were established through the following: 

• the use of multiple sources of data 

• the use of multiple methods of data collection 

• the use of multiple perspectives on the central evaluation objectives 

As stated above, standard techniques were employed to ensure the validity and reliability 
of the questionnaires and interviews. 



3.5 Ethical Procedures 

Ethical guidelines were followed at all times during the planning, implementing, analysing 
and reporting phases of thisevaluation. As a principle of research ethics, the design of 
this external evaluation consciously avoided any use of personal information on 
individuals that those individuals had, purposefully or not, provided by communicating 
electronically with the NRICH website or with any of the NRICH team. 

The internet is well-known as an open system. All forms of internet communication, be it 
web browsing or e-mail, are logged somewhere or other. For example, just as web 
servers keep a log of web traffic, e-mail servers keep a store of all their e-mail traffic. 
Thus it is theoretically possible to trawl through the e-mail correspondence on any of the 
NRICH e- mail facilities. This includes the ‘one-to-one’ service (where pupils can discuss 
any mathematical problem, ‘one-to-one’, with someone studying mathematics at 
university) and ‘NRICHtalk’ where pupils can discuss mathematics with other pupils. In 
the design of this evaluation the position was taken that such correspondence was private 
to those taking part and therefore should not be used as a source of data for the 
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evaluation under any circumstances. Similarly, although counts were made of 
registrations with NRICH (both by teachers and by pupils), the personal information 
provided by these individuals was never part of the evaluation. 

Ethical principles of confidentiality, informed consent, and no harm to the individual 
were maintained during the implementation phase of the evaluation. Data sifting and 
analysis followed standard procedures to ensure no bias was inadvertently introduced 
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4. Website Impact 

4.1 Introduction 

This segment is the largest in this report. It presents the analysis of the responses to the 
three questionnaires addressed to users of the NRICH website: pupils or students, school 
or college teachers, and other interested people (such as parents, University lecturers, 
local education authority employees, etc). Each of these groups of people is considered 
in turn, beginning with the pupil perspective. Detailed results are presented for each 
group. Summaries can be found at the end of each section (see sections 4.2.5, 4.3.5, and 
4.4.3). An overall summary is given in section 4.6. 

4.2 The Pupil Perspective 

A total of 513 responses to the pupil questionnaire were received. As noted in section 
3.3.1, close scrutiny was paid to the data in order to sift out, from the data for analysis, 
incomplete responses, unacceptable or frivolous responses, and multiple submissions 
from the same respondent. It was anticipated that a considerable number of incomplete 
responses to the questionnaires would be received. The close inspection of the. original 
data was designed to improve the validity of the data set used for analysis and hence the 
reliability of any conclusions drawn from the analysis. The result of this process are given 
below in Table 3, showing that 199 responses were finally accepted for analysis. 



Table 3: Responses to the pupil questionnaire 



total number of responses to pupil questionnaire 


513 


unspecified in every field 


95 


unspecified in almost every field (at the most, three answers specified) 


156 


added a comment that they had just found the site 


17 


clearly frivolous respondents (typically only a few responses given and these 
clearly not serious) 


26 


non-pupils (eq student teachers) 


20 


number of pupil responses accepted for analysis 


199 



The analysis that follows examines the following: 

• country of domicile of pupils 

• pupil gender and ethnicity 

• pupil age 

• type of school 

• where they access NRICH from 

• how often they make use of the various NRICH facilities 

• what they think of NRICH and its facilities 



The results are presented in some detail. A summary is provided in section 4.2.5. 
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4.2.1 About the pupils 

Just over half of the pupils who answered the questionnaire were from England (Table 
4). Almost 57% of pupil respondents were from the UK as a whole. Australian pupils 
were the next highest category (over 95% of them attending private schools 7 ). 



Table 4: Distribution of pupil respondents by country 



England 


102 


51.26% 


Australia 


38 


19.10% 


Singapore 


13 


6.53% 


USA 


7 


3.52% 


Scotland 


6 


3.02% 


Hong Kong 


5 


2.51% 


Wales 


5 


2.51% 


India 


3 


1.51% 


The Republic of Congo 


2 


1.01% 


Argentina 


1 


0.50% 


Bangladesh 


1 


0.50% 


Brunei 


1 


0.50% 


Italy 


1 


0.50% 


Kuwait 


1 


0.50% 


Norway 


1 


0.50% 


Turkey 


1 


0.50% 


not specified 


11 


5.53% 



7 According to Anderson (1993), private school enrolments in Australia are proportionally higher than 
any other Anglo-American country in the world and are growing, while elsewhere in the world they are 
static or declining. 

Anderson, D (1993), Public Schools in Decline: implications of the privatization of schools in Australia, 
in H Beare and W Lowe Boyd (Eds), Restructuring Schools. Falmer: London. 
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Over two-thirds of all the pupils who answered the pupil questionnaire were boys (Table 
5). Almost two-thirds of all the pupil respondents were white. The next highest ethnic 
group was Chinese. Around a third (1 1) of the Chinese pupils came from Singapore. The 
next largest group of Chinese pupils (8) came from Australia. Almost half of all the 
pupils who responded to the questionnaire were white boys. The next highest category 
was white girls at 19. 1%. 



Table 5: Gender and ethnicity of pupil respondents 





boy 

/male 


girl 

/female 


not 

specified 


Total 


White 


91 


38 


1 


130 

(65.3%) 


Chinese 


25 


6 


0 


31 

(15.6%) 


Other Asian Group 


8 


0 


0 


8 

(4.0%) 


Indian 


4 


3 


0 


7 

(3.5%) 


Mixed Race 


3 


1 


0 


4 

(2.0%) 


Black - African 


1 


3 


0 


4 

(2.0%) 


Other groups 


2 


1 


0 


3 

(1.5%) 


Bangladesh 


0 


3 


0 


3 

(1.5%) 


Black - Caribbean 


2 


0 


0 


2 

(1.0%) 


Pakistani 


0 


1 


0 


1 

(0.5%) 


unspecified 


0 


0 


6 


6 

(3.0%) 


Total 


136 

(68.3%) 


56 

(28.1%) 


7 

(3.5%) 


199 

(100%) 



Considering UK pupils (Table 6), 58% of pupil respondents were boys compared to 38% 
girls. Girls make up a greater proportion of the UK pupil respondents than that of all 
pupil respondents (38% compared to 28%), but the proportion still falls below that of 
girls in the school pupil age range in the UK. 

Just over 80% of all the UK pupil respondents who completed the pupil questionnaire 
gave their ethnicity as white (Table 5). A small number of respondents (4.5%) did not 
specify their ethnicity. Of those UK pupils who did specify their ethnicity (107 pupils), 
84% were white. This proportion is in line with national UK statistics on ethnic groups in 
formal education. The UK Department for Education and Employment figures on pupil 
ethnicity in England, published in June 1999, showed that 88.2% of primary aged pupils 
and 88.5% of secondary age pupil were white (DfEE 1999). The numbers of UK pupil 
respondents to the NRICH pupil questionnaire from other ethnic groups may suggest 
that there is a possible over-representation of certain groups (such as Chinese and other 
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Asian), coupled with a possible under-representation of other ethnic groups, such as 
black Caribbean heritage and Pakistani pupils. However, the numbers of respondents is 
certainly far too small to draw any firm conclusions on this point. 

The largest single category of UK pupil respondent was white boys (just over 48% of all 
UK pupil respondents). The second highest category was white girls at just over 3 1%. 



Table 6: Gender and ethnicity of pupil respondents from the UK 





boy 

/male 


girl 

/female 


not 

specified 


Total 

NRICH 


NRICH 

specified 


Ethnicity 

primary 8 


Ethnicity 

secondar 

y 


Ethnicity of 
UK 

population 9 


White 


54 


35 


1 


90 

(80.4%) 


84.1% 


88.2% 


88.5% 


94.5% 


Other Asian 
Group 


6 


0 


0 


6 

(5.4%) 


5.6% 


2.1% 


1.8% 


0.4% 


Chinese 


1 


2 


0 


3 

(2.7%) 


2.8% 


0.3% 


0.4% 


0.3% 


Indian 


1 


1 


0 


2 

(1 .8%) 


1.9% 


2.3% 


2.7% 


1.5% 


Mixed Race 


1 


1 


0 


2 

(1.8%) 


1.9% 


- 


- 


0.5% 


Bangladeshi 


0 


1 


0 


1 

(0.9%) 


0.9% 


1.0% 


0.9% 


0.3% 


Black - 
African 


0 


1 


0 


1 

(0.9%) 


0.9% 


1.2% 


1.0% 


0.4% 


Black - 
Caribbean 


1 


0 


0 


1 

(0.9%) 


0.9% 


1.6% 


1.4% 


0.9% 


Other 

groups 


1 


0 


0 


1 

(0.9%) 


0.9% 


0.9% 


0.7% 


0.3% 


Pakistani 


0 


1 


0 


1 

(0.9%) 


0.9% 


2.5% 


2.6% 


0.9% 


unspecified 


0 


0 


4 


4 

(3.6%) 


n/a 


n/a 


n/a 


n/a 


Total 


65 

(58.0%) 


42 

(37.5%) 


5 

(4.5%) 


112 

(100%) 


107 

(100%) 


100% 


100% 


100% 



8 Source: DfEE Statistics report sfrl5, Minority Ethnic Pupils in Maintained Schools by Local 
Education Authority Area in England: January 1999 (Provisional). 30 June 1999. 

9 Source: Owen, D (1992-1995), 1991 Census Statistical Papers 1-9, Centre for Research in Ethnic 
Relations, University of Warwick/CRE. 
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Almost three quarters of the pupils who answered the questionnaire were of secondary 
school age. The data is given in Table 7. Of the pupils aged 10 and under, almost two 
thirds were boys (12 out of 19). In the 1 1-16 age range, over 70% of the pupils were 
boys. The proportion of males to females in the group aged over 1 6 was similar to that 
of the 11-16 range. 



Table 7: Age distribution of pupil respondents 



age 10 or under 


19 


9.5% 


11-16 


146 


73.4% 


age over 16 


30 


15.1% 


not specified 


4 


2% 


Total 


199 


100% 



The students attended a range of institutions, with the largest number, although less than 
30% of the total, attending secondary comprehensive schools (Table 8). Pupils from 
private preparatory schools came a close second (34 of these pupils, over 70%, were 
from Australia). The data on the type of school is shown in Table 8. 



Table 8: Type of school attended by pupil respondents 



Secondary comprehensive 


56 


28.14% 


Private (preparatory) 


47 


23.62% 


Primary Gunior or infant) 


30 


15.08% 


16-19 college 


20 


10.05% 


Secondary selective 


20 


10.05% 


Private (upper) 


14 


7.04% 


Middle 


9 


4.52% 


5-17 private college 


1 


0.50% 


not specified 


2 


1.01% 


Total 


199 


100.00% 



Most of the pupils lived in a city (Table 9). 



Table 9: Type of area where pupil respondents lived 



in a large city 


116 


58.29% 


in a small town 


66 


33.17% 


in the countryside 


16 


8.04% 


unspecified 


1 


0.50% 


Total 


199 


100.00% 
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Most of the pupils who completed the questionnaire attended a mixed school, as shown 
in Table 10. Over half (57%) of those attending a boys school were from private 
preparatory schools in Australia, with just under twenty percent (12 pupils) from 
England, half of them from secondary selective schools. All the female pupil respondents 
attending a girls school, apart from one (who was from Bangladesh), were from England. 
The female English pupil respondents attending a girls school were evenly split between 
state and private education. The largest group (by 1) were those from private upper 
schools (5 respondents). No data on the socio-economic class of pupils was collected for 
this evaluation as such data is not very straightforward to collect reliably and feedback 
from the piloting of the pupil questionnaire suggested that it was too long for some 
pupils and that some questions should be omitted. 



Table 10: Gender mix of school attended by pupil respondents 



mixed school 


120 


60.30% 


boys school 


63 


31.66% 


girls school 


16 


8.04% 


Total 


199 


100.00% 



4.2.2 How pupils access NRICH 

In total, some 48% of pupils accessed NRICH at school, with most school access taking 
place in an IT room (Table 1 1). Almost exactly the same proportion of the pupils 
accessed NRICH at home. Fourteen pupils (7%) said they accessed NRICH from more 
than one place, almost all of them citing both school and home. There was no difference 
in the pattern of place of access between boys and girls. Very few of the pupil 
respondents accessed NRICH from a public library or other public access location. For 
those pupils accessing NRICH from home, the socio-economic status of the family 
allows not only the purchase of an internet-capable computer but also pays for internet 
provision and associated telephone costs. 



Table 11: Place of access to NRICH by pupil respondents 



at home 


95 


47.74% 


in the IT room 


50 


25.13% 


in my mathematics classroom 


32 


16.08% 


in the school library 


10 


5.03% 


in another classroom (not mathematics) 


4 


2.01% 


in the public library 


2 


1.01% 


In the Town Hall 


2 


1.01% 


at a university 


1 


0.50% 


at work with my mum 


1 


0.50% 


not specified 


2 


1.01% 


Total 


199 


100.00% 
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The ‘resource bank’ was the part of the NRICH site most used by the pupils, with almost 
25% of them saying that they access that part of the site “most weeks” (Table 12). A few 
pupils claimed to use some NRICH facilities almost every day, but for the majority it was 
less than once a month, if then. There was little difference between the relative usage of 
these parts of NRICH by girls and boys. 



Table 12: Pupil respondent use of the general NRICH facilities 





most 

days 


most 

weeks 


once a 
month 


less than 
once a 
month 


never 


not 

specified 


Total 


Ask NRICH 


11 

(5.5%) 


28 

(14%) 


25 

(12.6%) 


66 

(33.2%) 


60 

(30.1%) 


9 

(4.5%) 


199 

(100%) 


Emailing list 


10 

(5%) 


19 

(9.5%) 


25 

(12.6%) 


57 

(28.6%) 


74 

(37.2%) 


14 

(7%) 


199 

(100%) 


Games 


12 

(6%) 


37 

(18.6%) 


45 

(22.6%) 


49 

(24.6%) 


39 

(19.6%) 


17 

(8.5%) 


199 

(100%) 


Resource bank 


10 

(5%) 


49 

(24.6%) 


38 

(19.1%) 


45 

(22.6%) 


34 

(17.1%) 


23 

(11.6%) 


199 

(100%) 


Send solutions with 
teacher’s help 


6 

(3%) 


14 

(7%) 


18 

0%) 


55 

(27.6%) 


82 

(41.2%) 


24 

(1271%) 


199 

(100%) 


Send solutions on 
own 


13 

(6.5%) 


27 

(13.6%) 


26 

(13.1%) 


56 

(28.1%) 


53 

(26.6%) 


24 

(12.1%) 


199 

(100%) 



Where pupils sent in solutions, most did so on their own rather than with the assistance 
of a teacher (Table 12). Many pupils reported never sending in a solution, whether 
assisted by a teacher or not. Just over half of the pupils (52%) who said they sent in 
solutions on their own most weeks or most days accessed NRICH from their homes. 
Almost a third of this group of pupils (40 pupils), who said they sent in solutions that 
frequently, attended private preparatory schools, the largest single category. These 40 
pupils were almost evenly split between those from Australia, England and Singapore 
(just under a third from each of these three countries); 82.5% of them were boys. 
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Table 13 shows the data on frequency of usage by primary, middle and preparatory 
pupils of those elements of the NRICH site aimed at primary age pupils. All the different 
primary elements were accessed by pupils, with the first two problem sections in the 
table possibly having a very slight edge in popularity due to the age profile of the pupil 
respondents (‘Let me Try’ is primarily aimed at 5-8 years olds and there were few pupil 
respondents in that age range). 



Table 13: Usage of the primary NRICH facilities by primary pupils 





most 

days 


most 

weeks 


once a 
month 


less than 
once a 
month 


never 


not 

specified 


Total 


Bernard's Bag 
(open problems) 


3 

(3.5%) 


17 

(19.8%) 


15 

(17.4%) 


21 

(24.4%) 


25 

(29.1%) 


5 

(5.8%) 


86 

(100%) 


Penta Problems 


5 

(5.8%) 


15 

(17.4%) 


15 

(17.4%) 


16 

(18.6%) 


29 

(33.7%) 


6 

(7.0%) 


86 

(100%) 


Let Me Try 


4 

(4.7%) 


13 

(15.1%) 


11 

(12.8%) 


18 

(20.9%) 


30 

(34.9%) 


10 

(11.6%) 


86 

(100%) 


Kids Mag 


5 

(5.8%) 


14 

(16.3%) 


8 

(9.3%) 


23 

(26.7%) 


30 

(34.9%) 


6 

(7.0%) 


86 

(100%) 



Usage by secondary pupils of the sections of the NRICH aimed at that age of pupil is 
given in Table 14. There was not a great deal of difference in the relative popularity of 
the various elements in the table, although the ‘monthly problems’ and the ‘tough nuts’ 
were especially popular with pupils from selective schools in England, with pupils from 
English secondary comprehensive schools only just behind. Just over 73% of the pupils 
who said that they accessed the ‘tough nuts’ that often were boys. The corresponding 
figure for the monthly problems was 80%. Girls, in general, said they accessed the 
‘monthly problems’ and ‘tough nuts’ rather less frequently (usually less than once a 
month) compared with boys. 



Table 14: Usage of the secondary NRICH facilities by secondary pupils 





most 

days 


most 

weeks 


once a 
month 


less than 
once a 
month 


never 


not 

specified 


Total 


News 


5 

5.6% 


16 

17.8% 


14 

15.6% 


27 

30.0% 


21 

23.3% 


7 

7.8% 


90 

100.0% 


Articles 


4 

4.4% 


16 

17.8% 


18 

20.0% 


27 

30.0% 


16 

17.8% 


9 

10.0% 


90 

100.0% 


Monthly Six 
Problems 


8 

8.9% 


13 

14.4% 


20 

22.2% 


25 

27.8% 


16 

17.8% 


8 

8.9% 


90 

100.0% 


More Challenging 

Problems/Tough 

nuts 


9 

10.0% 


17 

18.9% 


14 

15.6% 


27 

30.0% 


14 

15.6% 


9 

10.0% 


90 

100.0% 
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4.2.3 What pupils think of NRICH 

Almost two-thirds of pupils agreed or strongly agreed that the NRICH website was well- 
designed, although around 25% claimed to find the website difficult to use (Table 15). 
There were mixed views on the ease of use of ‘askNRICH’. 



Table 15: Pupil respondents’ views of the NRICH website 





Strongly 

agree 


Agree 


Neither 


Disagree 


Strongly i 
disagree 


Don't 

know 


not 

specified 


Total 


NRICH website 
is well-designed 


36 

(18.1%) 


91 

(45.7%) 


28 

(14.1%) 


10 

(5%) 


14 

(7%) 


7 

(3.5%) 


13 

(6.5%) 


199 

(100%) 


NRICH is 
difficult to use 


11 

(5.5%) 


39 

(19.6%) 


37 

(18.6%) 


67 

(33.7%) 


19 

(9.5%) 


12 

(6%) 


14 

(7%) 


199 

(100%) 


ask NRICH is 
difficult to use 


16 

(8%) 


18 

(9%) 


55 

(27.6%) 


33 

(16.6%) 


17 

(8.5%) 


43 

(21.6%) 


17 

(8.5%) 


199 

(100%) 



Pupils have positive views about the NRICH facilities, although many of the pupil 
respondents (20%) were unable to express a view about ‘one-to-one’ (Table 16). Almost 
half the pupil respondents (47.3%) agreed or strongly agreed that NRICH made them 
feel part of a club. A similar proportion looked forward to each monthly edition. The 
articles got a mixed reception, but the problems and the games were well-liked. Seeing 
their solutions published was popular with a large proportion of pupils (half of all the 
pupils agreed or strongly agreed that they liked seeing their solutions published). 



Table 16: Pupil respondents’ views of the NRICH website 





Strongly 

agree 


Agree 


Neither 


Disagree 


Strongly 

disagree 


Don't 

know 


not 

specified 


Total 


One-to-One (ask 
a mathematician) 
is the best facility 


23 

(11.6%) 


35 

(17.6%) 


53 

(26.6%) 


17 

(8.5%) 


14 

(7%) 


40 

(20.1%) 


17 

(8.5%) 


199 

(100%) 


NRICH makes me 
feel part of a 
mathematics club 


29 

(14.6%) 


65 

(32.7%) 


37 

(18.6%) 


22 

(11.1%) 


20 

(10.1%) 


10 

(5%) 


16" 

(8%) 


199 

(100%) 


1 look forward to 
new NRICH 


30 

(15.1%) 


64 

(32.2%) 


39 

(19.6%) 


20 

(10.1%) 


18 

(9%) 


12 

(6%) 


16 

(8%) 


199 

(100%) 


1 never read the 
articles 


23 

(11.6%) 


31 

(15.6%) 


45 

(22.6%) 


50 

(25.1%) 


23 

(11.6%) 


9 

(4.5%) 


18 

1 (9%) 


199 

(100%) 


The problems are 
the best 


56 

(28.1%) 


64 

(32.7%) 


36 

(18.1%) 


8 

(4%) 


13 

(6.5%) 


7 

(3.5%) 


15 

(7.5%) 


199 

(100%) 


1 don’t understand 
why games are 
included 


10 

(5%) 


12 

(6%) 


29 

(14.6%) 


40 

(20.1%) 


81 

(40.7%) 


10 

(5%) 


17 

(8.5%) 


199 

(100%) 


1 like seeing my 

solutions 

published 


74 

(37.2%) 


23 

(11.6%) 


43 

(21.6%) 


4 

(2%) 


12 

(6%) 


23 

(11.6%) 


20 

(10.1%) 


199 

(100%) 
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Most pupils were referred to NRICH by their teacher, with a majority of those 
expressing an opinion indicating that NRICH was better than the mathematics they did in 
school (Table 17). Few pupils said they were at a school where there was a ‘maths club’ 
that used NRICH, although more pupils used NRICH at school than said that they did 
not (44% compared to 36%). Most of the pupil respondents claimed to find mathematics 
easy and did not think that the NRICH problems were too hard. Some pupils were able 
to work with friends on the NRICH problems (25%), but more did not (41%). 



Table 17: Pupil respondents’ views on aspects of the NRICH website 





Strongly 

agree 


Agree 


Neither 


Disagree 


Strongly 

disagree 


Don't 

know 


not 

specifie 

d 


Total 


My teacher 
suggested that 1 try 
NRICH 


57 

(28.6%) 


59 

(29.6%) 


18 

(9%) 


22 

(11.1%) 


28 

(14.1%) 


4 

(2%) 


11 

(5.5%) 


199 

(100%) 


NRICH is better 
than the maths 1 do 
at school 


45 

(22.6%) 


49 

(24.6%) 


49 

(24.6%) 


20 

(10.1%) 


13 

(6.5%) 


11 

(5.5%) 


12 

(6%) 


199 

(100%) 


at the school maths 
club we always use 
NRICH 


14 

(7%) 


17 

(8.5%) 


24 

(12.1%) 


53 

(26.6%) 


61 

(30.7%) 


18 

(9%) 


12 

(6%) 


199 

(100%) 


1 usually find 
mathematics easy 


52 

(26.1%) 


59 

(29.6%) 


34 

(17.1%) 


21 

(10.6%) 


14 

(7%1 


5 

(2.5%) 


14 

(7%) 


199 

(100%) 


1 never use NRICH 
in school 


37 

(18.6%) 


35 

(17.6%) 


25 

(12.6%) 


33 

(16.6%) 


54 

(27.1%) 


1 

(0.5%) 


14 

(7%) 


199 

(100%) 


1 always work with 
a friend on NRICH 


17 

(8.5%) 


33 

(16.6%) 


51 

(25.6%) 


34 

(17.1%) 


42 

(21.1%) 


5 

(2.5%) 


17 

(8.5%) 


199 

(100%) 


all the NRICH 
problems too hard 


12 

(6%) 


29 

(14.6%) 


49 

(24.6%) 


52 

(26.1%) 


31 

(15.6%) 


11 

(7%) 


15 

(7.5%) 


199 

(100%) 
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A majority of pupils thought that NRICH had made them more interested in mathematics 
and more likely to continue studying mathematics (Table 18). Opinion was more divided 
on whether friends think they are mad to like NRICH and whether or not a print version 
of NRICH would be preferable to the web-based version. 



Table 18: Pupil respondents’ views of the impact of NRICH website 





Strongly 

agree 


Agree 


Neither 


Disagree 


Strongly 

disagree 


Don't 

know 


not 

specifie 

d 


Total 


NRICH has made 
me more interested 
in mathematics 


31 

(15.6%) 


62 

(31.2%) 


40 

(20.1%) 


20 

(10.1%) 


20 

(10.1%) 


10 

(5%) 


16 

(8%) 


199 

(100%) 


my friends think 1 
am mad to like 
NRICH 


26 

(13.1%) 


28 

(14.1%) 


40 

(20.1%) 


25 

(12.6%) 


37 

(18.6%) 


23 

(11.6%) 


20 

(10.1%) 


199 

(100%) 


using NRICH has 
made me want to 
continue studying 
mathematics 


34 

(17.1%) 


49 

(24.6%) 


45 

(22.6%) 


23 

(11.6%) 


25 

(12.6%) 


8 

(4%) 


15 

(7.5%) 


199 

(100%) 


1 would prefer 
NRICH to be a 
printed magazine 


36 

(18.1%) 


33 

(16.6%) 


39 

(19.6%) 


29 

(14.6%) 


34 

(17.1%) 


T1 

(7%) 


17 

(8.5%) 


199 

(100%) 



Finally, only 20 of the pupil respondents (10% of all those responding) said that they 
were registered with NRICH (Table 19). Given that by October 1999, a total of more 
than 1300 pupils had registered with NRICH (cumulative registrations in the period 
October 1997-October 1999; data supplied by NRICH), this low number is somewhat 
surprising. There are a number of possible explanations. Amongst these are that: 

• many of the respondents chose not to reveal that they were registered (one example of 
this was uncovered by the follow-up e-mail to some 70 pupil respondents even though 
only four replies were received), 

• many of the registered pupils no longer access the NRICH site 

• while the NRICH site provides a straightforward method of registering, there is no 
mention of how to de-register 

These last two points could suggest that the cumulative number of pupils registered with 
NRICH may not be not a reliable indicator of how many pupils, and of what type, access 
the NRICH website. 

Table 19: Number of pupil respondents registered with NRICH 



Registered with NRICH 




yes 


20 

(10.1%) 


not specified 


179 

(89.9%) 


Totals 


199 

(100%) 
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The low number of pupil respondents who were registered with NRICH had a knock-on 
effect on another component of the evaluation, the case profiles of pupils who make use 
of the wider NRICH facilities such as the e-mail answering service. As described in 
section 3.2.4, the effect was to severely limit the number of pupils who could be 
contacted to provide further information on their experience of using NRICH. This was 
not anticipated at the planning stage of this evaluation, as, with a planned focus on the 
regular NRICH user, a higher number of registered pupils were expected to complete the 
questionnaire. 

4.2.4 Pupils’ comments on NRICH 

Just under a quarter of the pupil respondents added a comment to their questionnaire (43 
respondents out of 199). Almost every comment was complimentary. Some were short 
(“Excellent! !”, “I like it lots”). Many other were quite long. 

Below are some typical examples of complimentary comments: 

“All the Nrich problems are challenging to me, not like the problems I do at 
school. Doing the problems has made me feel how it feels to be stuck on a 
mathematics problem and do not know how to do it. ’’ 

1 1 year old boy from Singapore. 

“I really like this site and think it is great, although my friends don't like 
maths that much I am trying to wean them onto this site because it is really 
interesting and helps me with my maths. ” 

12 year old girl from England. 

“I think N-rich is really cool and has made me think differently about 
mathematics. ” 

12 year old boy from the USA. 

“I use Nrich every week in our maths club. It's a great site! ” 

13 year old boy from England. 

“Well, it’s better than school maths!” 

12 year old boy from a private preparatory school. 

“NRICH is cool. It does not just do the simple types of mathematics but it does 
problem solving aswell. It is really fun and is great to log onto during breaks. ” 

12 year old girl from a private upper school in England. 

" The problems are often challenging, but are still good fun. ” 

1 1 year old boy from an Australian private preparatory school. 

“Thank you for your help. As an [adult] outreach student, it is hard to find 
help for math problems when the teacher is too busy to help. ” 

41 year old female student at an American college in a small town. 

Some pupils chose to make suggestions. Below are some typical examples: 

“I think the problems should be in three different groups. Easy, Medium, 

Hard ” 

12 year old boy. 
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“It is great fun and my class love it. It is really easy to use but I wish the 
games were changed more often. ” 

12 year old girl from an English private upper school. 

“It would be good if the NRICH site could give you a demonstration on a 
problem you may not know what sort of technique to use. ” 

1 1 year old boy from an Australian private preparatory school. 

“The nrich graphics are bad otherwise it is good ” 

1 1 year old boy 

“Needs to have more colour, it is a bit drab and boring. I mean, we are 
mathematicians but we do have style. ” 

1 1 year old boy from a private upper school in Wales. 

From two Australian private schools came the following comments: 

“I only use nrich because I have to. ” 

12 year old boy from an Australian private preparatory school. 

“I really don ’t like maths at school but I am in the A group (top) so we use 
NRICH once a week. It’s better than doing other written maths. ” 

A different 12 year old boy from an Australian private preparatory school. 

The above pupil comments illustrate how important NRICH was to many of the pupils 
who accessed the site. The suggestions to improve the NRICH facilities reflect a mature 
consideration of the site and fit with some of the outcomes of other components of this 
external evaluation. For instance, some of the pupils interviewed for the case studies, 
reported in section 6 of this report, and for the case profiles, reported in section 7 of this 
report, commented that more instant feedback on their solutions (like that provided by 
some forms of mathematics software packages that they use, for example) would be very 
helpful. Likewise, the website evaluation, reported in the next section of this report, 
found an improvement in the design of the site following the new layout launched in July 
1999. 

A few of the comments from pupil respondents indicated that NRICH is, in some cases, 
becoming part of the scheme of work that pupils have to follow, presumably in 
mathematics lessons, but possibly as part of an Information Technology course. This may 
be something that increases as more schools come ‘online’ and curriculum managers seek 
internet-based activities for courses of various kinds. 



4.2.5 Summary of the pupil perspective 



Almost 60% of the pupils who answered the questionnaire lived in the United Kingdom, 
and almost all of them in England. Australian pupils were the next highest category at 
just under 20%. All but two of the pupil respondents from Australia attended private 
schools. 

Over two-thirds of all the pupils who answered the pupil questionnaire were boys. 
Almost the same proportion were white. The next largest ethnic group (at just over 15% 
of the pupils) was Chinese, with around a third of these pupils coming from Singapore. 

Of UK pupils, 58% were boys compared to 38% girls. This proportion of girls is below 
that of girls in the school pupil age range in the UK. About 84% of UK pupils who 
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responded to the questionnaire were white, compared to a school population that is 
about 88% white. While the number of respondents in different categories was far too 
small to draw any firm conclusions, it is possible that there is some over-representation 
of certain groups of UK pupils (such as Chinese and other Asian), along with some 
possible under-representation of other groups, such as girls (in general), pupils of black 
Caribbean heritage, and Pakistani pupils. No data on the socio-economic class of pupils 
was collected as part of this evaluation. 

Almost three quarters of the pupils who answered the questionnaire were of secondary 
school age. Most lived in a city and attended a mixed school. The students attended a 
range of institutions, with the largest number, although less than 30% of the total, 
attending secondary comprehensive schools. Pupils from private preparatory schools 
came a close second (over 70% of such pupils were from Australia). 

Just under half the pupils accessed NRICH at school, usually in an IT room; almost 
exactly the same proportion of pupils said they accessed NRICH at home. A few said 
they accessed NRICH from both school and home. There was no difference in the 
pattern of access between boys and girls. Very few of the pupil respondents accessed 
NRICH from a public library or other public access location. 

A few pupils claimed to use some NRICH facilities almost every day, but for the majority 
it was less than once a month. In both the primary and secondary parts of NRICH, it was 
the problems that most attracted the interest in pupils. Almost two-thirds of pupils 
agreed or strongly agreed that the NRICH website was well-designed, although around 
25% claimed to find the website difficult to use. 

Pupils had positive views about all the NRICH facilities, with almost half agreeing or 
strongly agreeing that NRICH made them feel part of a club. Seeing their solutions 
published was popular with a large proportion of pupils. 

Most pupils were referred to NRICH by their teacher. Quite a number thought that 
NRICH was better than the mathematics they did in school. Some pupils were able to 
work with friends on the NRICH problems (25%), but many more did not (41%), 
perhaps reflecting the fact that many pupils access NRICH from home. A majority of 
pupils thought that NRICH had made them more interested in mathematics and more 
likely to continue studying mathematics. Only a small minority of the pupil respondents 
said that they were registered with NRICH. 

Just under a quarter of the pupil respondents added a comment to their questionnaire. 
Virtually every comment was positive and illustrated how important NRICH was to 
many of the pupils who accessed the site. 
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4.3 The Teacher Perspective 

A total of 999 responses to the teacher questionnaire were received. As noted in section 
3.3.1, close scrutiny was paid to the data in order to sift out, from the data for analysis, 
incomplete responses, unacceptable or frivolous responses, and multiple submissions 
from the same respondent. It was anticipated that a considerable number of incomplete 
responses to the questionnaires would be received. The close inspection of the original 
data was designed to improve the validity of the data set used for analysis and hence the 
reliability of any conclusions drawn from the analysis. The result of this process are given 
below in Table 20, showing that 450 teacher responses were finally accepted for analysis. 



Table 20: Responses to the teacher questionnaire 



total number of responses to the teacher questionnaire 


999 


unspecified in every field 


157 


unspecified in almost every field (at the most, three answers specified) 


392 


number of teacher responses accepted for analysis 


450 



The analysis that follows examines the following: 

• country of domicile of the teachers 

• type and phase of school, its status, location and number of pupils 

• the use made of NRICH by teachers 

• what teachers think of NRICH and its facilities 



The results are presented below in some detail. A summary is provided in section 4.3.5. 
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4.3.1 About the teachers 

The overwhelming majority of teachers who answered the questionnaire were from 
England (Table 21). 82% of teacher respondents were from the UK as a whole. US 
teachers were the next highest category, but at less than 5% of the total. 



Table 21: Country of domicile of teacher respondents 



school country 


Total 


% 


England 


343 


76.22% 


USA 


22 


4.89% 


Australia 


15 


3.33% 


Scotland 


14 


3.11% 


Wales 


8 


1.78% 


New Zealand 


5 


1.11% 


Canada 


4 


0.89% 


Northern Ireland 


4 


0.89% 


Singapore 


4 


0.89% 


Malaysia 


2 


0.44% 


Switzerland 


2 


0.44% 


Belgium 


1 


0.22% 


Cyprus 


1 


0.22% 


Finland 


1 


0.22% 


Hong Kong 


1 


0.22% 


Ireland 


7 


0.22% 


Isle of Man 


i 


0.22% 


Italy 


i 


0.22% 


Japan 


i 


0.22% 


Jersey 


i 


0.22% 


Mexico 


i 


0.22% 


Norway 


i 


0.22% 


Pakistan 


i 


0.22% 


Philippines 


i 


0.22% 


Saudi Arabia 


i 


0.22% 


Sweden 


i 


0.22% 


United Arab Emirates 


i 


0.22% 


not specified 


ii 


2.44% 


Total 


450 


100.00% 
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Around a third of the teacher respondents worked in secondary comprehensive schools 
(Table 22). About the same proportion were from primary (junior or infant) schools. 



Table 22: Type of school where teacher respondents taught 



Secondary comprehensive 


153 


34.00% 


Primary (junior or infant) 


152 


33.78% 


Middle 


29 


6.44% 


Private (upper) 


29 


6.44% 


Secondary selective 


20 


4.44% 


16-19 college 


17 


3.78% 


Private (preparatory) 


13 


2.89% 


University Initial Teacher Education 


6 


1.33% 


F E college 


5 


1.11% 


Special 


4 


0.89% 


13-18 comprehensive 


3 


0.67% 


Private international 5-18 


2 


0.44% 


Secondary Modem 


2 


0.44% 


14-18 Selective 


1 


0.22% 


graduate school 


1 


0.22% 


homeschool 


1 


0.22% 


kindergarten 


1 


0.22% 


Local Education Authority 


1 


0.22% 


Private 3-16 


1 


0.22% 


Private College 


1 


0.22% 


Private internet school 


1 


0.22% 


Private Secondary 


1 


0.22% 


not specified 


6 


1.33% 


Total 


450 


100.00% 



More than half the teachers worked in a school run by a local education authority (Table 
23). Around 10% were from private schools (over 70% of which were located in 
England). 



Table 23: Employer of teacher respondents 



Local Education Authority (state) school 


242 


53.78% 


Private school 


45 


10.00% 


Grant maintained/foundation school 


36 


8.00% 


Voluntary aided or controlled (Church) school 


31 


6.89% 


Further education college 


14 


3.11% 


self-employed (primary supply teacher) 


1 


0.22% 


not specified 


81 


18.00% 


Total 


450 


100.00% 
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Most of the teachers worked in a suburban school (Table 24). About half the teachers 
who worked in inner city schools (29 out of 67) worked in primary schools. 



Table 24: Location of teacher respondents’ schools 



Suburban 


153 


34.00% 


Urban 


127 


28.22% 


Rural 


95 


21.11% 


Inner-city 


67 


14.89% 


not specified 


8 


1.78% 


Total 


450 


100.00% 



The primary teachers tended to work in schools of between 200 and 499 pupils while the 
secondary teachers were evenly split between those who worked in schools of under 
1000 pupils arid those who worked in schools of over 1000 pupils (Table 25). 



Table 25: Size of teacher respondents’ schools 





LEA 


Private 


Grant 

maintained 

/foundation 


Voluntary 
aided or 
controlled 


Further 

education 


Primary 

Supply 

Teacher 


not 

specified 


Total 


200 - 499 


81 


16 


2 


17 


1 


1 


21 


139 

(30.0%) 


1000 
or more 


62 


3 


17 


2 


10 


0 


21 


115 

(25.6%) 


500 - 999 


57 


15 


14 


4 


2 


0 


22 


114 

(25.3%) 


less than 
200 


26 


9 


0 


6 


0 


0 


10 


51 

(11.3%) 


not 

specified 


16 


2 


3 


2 


1 


0 


7 


31 

(6.9%) 


Total 


242 


45 


36 


31 


14 


1 


81 


450 

(100%) 



The overwhelming proportion of teachers worked in mixed gender schools (Table 26). 



Table 26: Pupil gender mix in teacher respondents’ schools 



Mixed (all years) 


391 


86.89% 


Girls 


25 


5.56% 


Boys 


22 


4.89% 


Mixed (some years) 


5 


1.11% 


not specified 


7 


1.56% 


Total 


450 


100.00% 
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Many teachers (almost 30% of the respondents) had found NRICH through browsing 
the web (Table 27) . Just under 20% had learnt about NRICH from colleagues. Articles 
in journals and magazines, and inservice courses were also major sources. 



Table 27 : How the teachers had leamt about NRICH 



from browsing the web 


130 


28.89% 


from a colleague 


84 


18.67% 


from an article in a journal 


79 


17.56% 


from a leaflet 


65 


14.44% 


from an INSET course 


52 


11.56% 


direct from NRICH personnel 


6 


1.33% 


from the pilot project in Norfolk 


5 


1.11% 


from a pupil 


3 


0.67% 


from the National Numeracy Project 


3 


0.67% 


from a professional association 


3 


0.67% 


from a University lecturer 


2 


0.44% 


from a poster 


1 


0.22% 


from an Exhibition 


1 


0.22% 


from a Seminar 


1 


0.22% 


not specified 


15 


3.33% 


Total 


450 


100.00% 



Just over 60% of the teachers were relatively new to NRICH, having only been accessing 
the website for six months or less (Table 28). 

Table 28 : Length of time teacher respondents’ had been accessing NRICH 



a month 


179 


39.78% 


six months 


110 


24.44% 


a year 


58 


12.89% 


a year or two 


54 


12.00% 


more than two years 


10 


2.22% 


not specified 


39 


8.67% 


Total 


450 


100.00% 
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4.3.2 How teachers access NRICH 

The overwhelmingly majority of teachers were accessing NRICH from home (Table 29). 
Relatively few accessed NRICH from their own use in their classroom. Around a tenth 
reported accessing NRICH from more than one place, usually both home and school. 



Table 29: Where teacher respondents’ access NRICH 



home 


276 


61.33% 


IT suite 


49 


10.89% 


departmental office/room 


44 


9.78% 


school staffroom 


31 


6.89% 


school library 


15 


3.33% 


classroom 


8 


1.78% 


laptop 


1 


0.22% 


university 


1 


0.22% 


not specified 


25 


5.56% 


Total 


450 


100.00% 



The most used general NRICH facility was the resource bank of problems (Table 30). 
Sizeable proportions of teachers (up to or over 40%) said that they never used ‘ask 
NRICH’, or any of the bulletin boards, or sent in pupils’ solutions. Around half of the 
teachers who said that they never used these facilities were relatively new to NRICH, as 
were around the same proportion who said they accessed these sections of NRICH either 
most days or most weeks. 



Table 30: Teacher respondents’ usage of the general NRICH facilities 





most 

days 


most 

weeks 


once a 
month 


less than 
once a 
month 


never 


not 

specified 


Total 


ask NRICH 


9 

(2.0%) 


36 

(8.0%) 


52 

(11.6%) 


124 

(27.6%) 


178 

(39.6%) 


51 

(11.3%) 


450 

(100%) 


e-mail bulletin 
boards 


6 

(1.3%) 


17 

(3.8%) 


17 

(3.8%) 


114 

(25.3%) 


214 

(47.6%) 


82 

(18.0%) 


450 

(100%) 


games 


9 

(2.0%) 


43 

(9.6%) 


125 

(27.8%) 


128 

(28.4%) 


67 

(14.9%) 


78 

(17.3%) 


450 

(100%) 


resource bank 
(of problems) 


8 

(1.8%) 


71 

(15.8%) 


107 

(23.7%) 


134 

(29.8%) 


50 

(11.1%) 


80 

(17.8%) 


450 

(100%) 


send in pupil 
solutions 


2 

(0.4%) 


8 

(1.8%) 


21 

(4.7%) 


91 

(20.2%) 


209 

(46.4%) 


119 

(26.4%) 


450 

(100%) 
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Table 3 1 shows the data on frequency of usage by primary, middle and preparatory 
school teachers of those elements of the NRICH site aimed at primary age pupils. All the 
elements were popular, with the problems having the slight edge. Most primary, middle 
and preparatory school teachers who completed the questionnaire accessed NRICH once 
a month or less. 



Table 31: Primary teachers’ use of the primary NRICH facilities 





most 

days 


most 

weeks 


once a 
month 


less than 
once a 
month 


never 


not 

specified 


Total 


Bernard's Bag 
(open problems) 


3 

(1.5%) 


31 

(16.0%) 


81 

(41.7%) 


58 

(29.9%) 


8 

(4.1%) 


13 

(6.7%) 


194 

(100%) 


Penta Problems 


3 

(1.5%) 


28 

(14.4%) 


76 

(39.2%) 


56 

(28.9%) 


11 

(5.6%) 


20 

(10.3%) 


194 

(100%) 


Let Me Try 


3 

(1.5%) 


32 

(16.5%) 


57 

(29.4%) 


56 

(28.9%) 


21 

(10.8%) 


25 

(12.9%) 


194 

(100%) 


Kids Mag 


2 

(1.0%) 


25 

(12.9%) 


54 

(27.8%) 


57 

(29.4%) 


29 

(14.9%) 


27 

(13.9%) 


194 

(100%) 


Primary 

Staffroom 


2 

(1.0%) 


19 

(7.8%) 


46 

(23.7%) 


60 

(30.9%) 


29 

(14.9%) 


38 

(19.6%> 


194 

(100%) 



Usage by secondary teachers of the sections of the NRICH aimed at secondary age 
pupils is given in Table 32. The ‘monthly problems’ and the ‘tough nuts’ were especially 
popular, with around two-thirds of all secondary teachers accessing them at least once a 
month. Around half of all secondary teachers also checked the news and the articles at 
least once a month. 



Table 32: Secondary teachers’ use of the secondary NRICH facilities 





most 

days 


most 

weeks 


once a 
month 


less than 
once a 
month 


never 


not 

specified 


Total 


Main news 


5 

(2.4%) 


22 

(10.5%) 


74 

(35.2%) 


45 

(21.4%) 


28 

(13.3%) 


36 

(17.1%) 


210 

(100%) 


Articles 


3 

(1.4%) 


30 

(14.3%) 


80 

(38.1%) 


48 

(22.9%) 


22 

(10.5%) 


27 

(12.9%) 


210 

(100%) 


Monthly 

problems 


7 

(3.3%) 


35 

(16.7%) 


109 

(51.9%) 


41 

(19.5%) 


9 

(4.3%) 


9 

(4.3%) 


210 

(100%) 


Challenging 
problems/ tough 
nuts 


5 

(2.4%) 


25 

(11.0%) 


105 

(50.0%) 


42 

(20.0%) 


12 

(5.7%) 


21 

(10.0%) 


210 

(100%) 
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Over 60% of all the teachers (both primary secondary) looked forward to each new 
NRICH issue (Table 33). The teachers were aware of most of the NRICH facilities. 
While two thirds agreed or strongly agreed that the problems were the best part of 
NRICH, half said the ‘Resource Bank’ was not the only part of NRICH they used, and a 
similar proportion disagreed (some strongly) that they never read the articles. 
Nevertheless, some 43% were unsure about ‘One-to-One’ (perhaps because it is aimed 
more at pupils and so they did not have personal experience of it), and more than a third 
agreed or strongly agreed that sending in pupil solutions was too time consuming. 



Table 33: Teachers’ views of NRICH facilities 





Strongly 

agree 


Agree 


Disagree 


Strongly 

disagree 


Don’t 

know 


not 

specified 


Total 


1 always look forward to 
new NRICH monthly 
editions 


49 

(10.8%) 


233 

(51.8%) 


40 

(8.9%) 


7 

(1.6%) 


59 

(13.1%) 


62 

(13.8%) 


450 

(100%) 


1 never read the articles 
on the main site 


13 

(2.19) 


87 

(19.3%) 


205 

(45.6%) 


42 

(9.3%) 


39 

(8.7%) 


64 

(14.2%) 


450 

(100%) 


The problems are the 
best part 


89 

(19.8%) 


212 

(47.1%) 


47 

(10.4%) 


3 

(0.7%) 


45 

(10.0%) 


54 

(12.0%) 


450 

(100%) 


Sending in solutions is 
too time consuming 


22 

(4.9%) 


144 

(32.0%) 


89 

(19.8%) 


14 

(3.1%) 


-105 

(23.3%) 


76 

(16.9%) 


450 

(100%) 


The Resource Bank is 
the only part of NRICH 1 
regularly use 


12 

(2.7%) 


98 

(21.8%) 


180 

(40.0%) 


44 

(9.8%) 


46 

(10.2%) 


70 

(15.6%) 


450 

(100%) 


1 don’t have the time to 
use the NRICH bulletin 
boards 


37 

(8.2%) 


196 

(43.6%) 


87 

(19.3%) 


6 

(1.3%) 


72 

(16.0%) 


52 

(11.6%) 


450 

(100%) 


One-to-One is the best 
facility provided by 
NRICH 


8 

(1.8%) 


62 

(13.8%) 


105 

(23.3%) 


14 

(3.1%) 


195 

(43.3%) 


66 

(14.7%) 


450 

(100%) 
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Two thirds of teachers said they mainly used NRICH as a source of problems for use in 
teaching with their classes, with few saying they mainly used NRICH for homework 
(Table 34). Almost half said that they had encouraged their pupils to access NRICH 
independently of school. Most teachers reported not having a Maths Club based around 
NRICH. 

The teachers did not only recommend NRICH to their more able pupils. A majority 
agreed (many strongly) that NRICH was particularly good for pupils who have a talent 
for mathematics. 



Table 34: Teachers’ use of NRICH facilities 





Strongly 

agree 


Agree 


Disagree 


Strongly 

disagree 


Don't 

know 


not 

specified 


Total 


1 mainly use NRICH as 
a source of problems 
to use in my teaching 


73 

(16.2%) 


230 

(51.1%) 


62 

(13.8%) 


3 

(0.7%) 


24 

(5.3%) 


58 

(12.9%) 


450 

(100%) 


1 never suggest to my 
pupils to access 
NRICH independently 


13 

(2.9 %) 


79 

(17.6%) 


182 

(40.4%) 


81 

(18.0%) 


25 

(5.6%) 


70 

(15.6%) 


450 

(100%) 


My school has a 
maths club based 
around the NRICH 
facilities 


14 

(3.1%) 


52 

(11.6%) 


196 

(43.6%) 


79 

(17.6%) 


28 

(6.2%) 


81 

(18.'0%) 


450 

(100%) 


1 only recommend 
NRICH to the most 
able pupils in my 
classes 


8 

(1.8%) 


85 

(18.9%) 


199 

(44.2%) 


49 

(10.9%) 


32 

(7.1%) 


77" 

(17.1%) 


450 

(100%) 


1 mostly use NRICH 
for setting homework 


4 

(0.8%) 


30 

(6.7%) 


223 

(51.8%) 


85 

(18.9%) 


28 

(6.2%) 


80 

(17.8%) 


450 

(100%) 


My pupils will never 
have heard of NRICH 


25 

(5.6%) 


101 

(22.4%) 


171 

(38.0%) 


35 

(7.8%) 


33 

(7.3%) 


85 

(18.9%) 


450 

(100%) 


NRICH is particularly 
good for those pupils 
of mine that have a 
talent for mathematics 


47 

(10.4%) 


203 

(45.1%) 


62 

(13.8%) 


13 

(28.9%) 


44 

(9.8%) 


81 

(18.0%) 


450 

(100%) 
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4.3.3 What teachers think of NRICH 

Almost 85% of teachers agreed or strongly agreed that the NRICH website was well- 
designed, with only around 10% claiming to find the website difficult to use (Table 35). 
Just over 40% agreed or strongly agreed that NRICH made them feel that they can share 
issues with other mathematics teachers, although over a quarter said “don’t know”. Few 
teachers thought NRICH would be better as a printed magazine. A majority of teachers 
thought that using NRICH had made their pupils more interested in mathematics, 
although around 20% said that they didn’t know. 



Table 35: Teachers’ view of the NRICH website 





Strongly 

agree 


Agree 


Disagree 


Strongly 

disagree 


Don’t 

know 


not 

specified 


Total 


the NRICH website is 
well-designed 


106 

(23.6%) 


269 

(59.8%) 


8 

(1.8%) 


2 

(0.4%) 


27 

(3.8%) 


38 

(8.4%) 


450 

(100%) 


it is difficult to find 
what you are looking 
for on NRICH website 


4 

(0.8%) 


42 

(9.3%) 


255 

(56.7%) 


88 

(19.6%) 


21 

(4.7%) 


40 

(8.9%) 


450 

(100%) 


NRICH makes me feel 
1 can share issues 
with other teachers 


13 

(2.9%) 


171 

(38.0%) 


66 

(14.7%) 


9 

(2.0%) 


125 

(27.8%) 


66 

(14.7%) 


450 

(100%) 


There is no advantage 
for NRICH to on the 
www. It would be 
better as a printed 
magazine. 


7 

(1.6%) 


24 

(5.3%) 


185" 

(41.1%) 


125 

(27.8%) 


32 

(7.1%) 


77 

(17.1%) 


450 

(100%) 


NRICH is just an 
entertaining pastime 


7 

(1.6%) 


38 

(8.4%) 


186 

(41.3%) 


108 

(24.0%) 


37 

(8.2%) 


74 


450 

(100%) 


Using NRICH with my 
pupils has made them 
more interested in 
mathematics 


37 

(8.2%) 


184 

(40.9%) 


39 

(8.7%) 


7 

(1.6%) 


98 

(21.8%) 


85 

(18.9%) 


450 

(100%) 



Finally, 23 of the teachers who completed the questionnaire said that they were 
registered with NRICH (see table 36), only 5% of all those responding. As with the 
results of the pupil questionnaire, this low proportion is a little surprising, particularly as 
between October 1997 and October 1999, a total of more than 1400 teachers had 
registered with NRICH (cumulative registrations in the period October 1997-October 
1999; data supplied by NRICH). There are a number of possible explanations. Amongst 
these are that: 

• many of the teacher respondents chose not to reveal that they were registered 

• many of the registered teachers no longer access the NRICH site 

• while the NRICH site provides a straightforward method of registering, there is no 
mention of how to de-register 



These last two points given above might indicate that the cumulative number of teachers 
registered with NRICH may not be not a reliable indicator of how many teachers access 
the NRICH website (and where from), nor how they use it. 
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Table 36: Number of teacher respondents registered with NRICH 



Registered with NRICH 






yes 


23 


5.1% 


not specified 


427 


94.9% 


Total 


450 


100% 



As with the low number of pupil respondents who said that they were registered with 
NRICH, the low number of registered teacher respondents had an impact on another 
element of the evaluation, the case studies of school and classroom usage of NRICH. As 
described in section 3.2.3, the effect was to severely limit the number of teachers who 
could be contacted with a view to gaining further information on their experience of 
using NRICH. This was not anticipated at the planning stage of this evaluation, as, with a 
planned focus on the regular NRICH user, a higher number of registered teachers were 
expected to complete the questionnaire. 

4.3.4 Teachers’ comments on NRICH 

About 15% of the teacher respondents added a comment to their questionnaire (65 
respondents out of 450). Almost every comment was complimentary. Some were short 
(“it’s really great”), others were quite long. 

Below are some typical examples of the longer complimentary comments: 

“ Thanks ! It even has me entertained with the puzzles. ” 

Teacher in an English secondary comprehensive school. 

“NRICH has increased my personal interest and enjoyment of maths as a 
teacher (and Maths coordinator). Iam about to start a maths club based on 
NRICH activities. ” 

Teacher in an English inner-city primary school. 

“I think this is a wonderful site and I only wish I had more time to access it. I 
use the problems with more able children who are withdrawn from the 
classroom all across the Junior age range, but I also use them with a Maths 
Club I ran that is open to every pupil. " 

Teacher in an English suburban primary school. 

Many teachers particularly commented on how they made use of NRICH. Below are 
some examples: 

“I download the problems pages once a month to put on the school intranet. I 
make pupils aware of its presence but I have no way of monitoring how 
frequently it is used I also print out problems from to time for use as 
additional problems for class use. ’’ 

Teacher from an English private preparatory school. 

“I use some of the Monthly Six for my Year 10 top set [highest attaining 14/15 
year olds] for occasional homeworks, as an alternative to homeworks from out 
of their textbooks. ’’ 

Teacher in an English secondary comprehensive school. 
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“I was using NRICH for Booster classes long before the Government came up 
with the idea!! ” 

Teacher in an English urban primary school. 

“Since discovering this site about 6 months ago I have downloaded problems 
and shared them with my students and my staff. Some of my staff have shared 
the problems with their students. Students have visited the site from their own 
computers. ” 

Teacher in an Australian secondary comprehensive school. 

“I use it to back up activities, or as an ‘extra fun ’ item. The children enjoy the 
challenges, whatever their ability. ” 

Teacher in an English rural primary school. 

“Mostly, as a teacher, I just refer students to the site. Some have used it and 
even sent in solutions ! Keep up the good work! ” 

Teacher in an English secondary comprehensive school. 

“The reason that I do not recommend NRICH to my pupils is that it is an 
important resource for me to use in the class. There are very few maths 
resource books that contain interesting (for the pupils) investigations. ” 

Teacher in an English suburban primary school. 

Many of the teachers commented on pupils reaction to NRICH. Below are some typical 
examples: 

“The more challenging exercises have gone down a storm. We will be using 
Bernard's Bag across the school next year! ” 

Teacher in an English suburban primary school. 

“I use it [NRICH] as a resource for class problem solving and investigation 
lessons, changing the problems as needed to suit the children. The children 
enjoy the problems, and are enthusiastic about Maths as a result. ” 

Teacher in an English rural primary school. 

“Students think of these problems as 'real' and interesting - not like doing real 
work! Brilliant - if that's what they think. I'm all for it, especially when it 
produces fantastic results! ” 

Teacher in a rural English secondary comprehensive school. 

“The children enjoy finding the problems on the internet and it gives their 
work an extra boost if they think they can send in their answers. ” 

Teacher in an English urban primary school. 

“The harder problems are personally rewarding to staff as well as students. I'd 
like to get my students to send in their answers, but they aren't always keen to 
write them up properly once they have solved the problem. ” 

Teacher in a suburban English secondary comprehensive school. 

“The problems have generated interest in Maths. ” 

Teacher in an English private upper school. 

Some of the teachers had suggestions for ways of developing aspects of NRICH. Some 
examples are given below: 
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“The monthly problems could perhaps be grouped into 2 problems for 
Primary ( 11-12 years old), 2 problems for Lower Secondary (13-14 years old) 
and 2 for Upper Secondary (15-16 years old). So that pupils of all ages can 
participate every month. ” 

Teacher in a primary school in Singapore. 

“I find MUCH to be a valuable resource of ideas, communication and 
resources delivered through the perfect medium, (and for many students a 
motivating medium). I would like to see, however, more material for the lower 
ability secondary students”. 

Teacher in an English secondary comprehensive school. 

“A fantastic service that should not just focus on the more able. You have a 
good model in NR1CH - 1 wish somebody would do as good a job for the whole 
range of abilities. ” 

Teacher in an English secondary comprehensive school. 

“More low achievers material would be appreciated ” 

Teacher in a rural English secondary comprehensive school. 

Some teachers took the opportunity to report difficulties they had experienced or 
limitations on their use of NRICH. Below are some typical examples (not counting those 
who reported only limited internet access): 

“My class love the puzzles etc and have tried to send in solutions only to find 
that you didn't receive them or they became scrambled - perhaps this could be 
made fool proof so children can do everything themselves and be successful 
and get a reply. ” 

Teacher in a Japanese private preparatory school. 

“I have attempted to register but have not received a reply. ” 

Teacher in an English secondary comprehensive school. 

“I use nrich spasmodically. I try and look at in during half-terms and 
holidays, otherwise I don't have the time. I find I don't have as much time 
within the curriculum as I would like to use some of the problems. ” 

Teacher in an English private school. 

“I think this is a wonderful site and I only wish I had more time to access it. ” 
Teacher in an English suburban primary school. 

“I would use this site more if I had quicker access to it in my classroom. ” 

Teacher in an American suburban primary school. 

The above teacher comments illustrate how important NRICH was to many of the 
teachers who responded to the questionnaire. The range of comments resonate with 
some of the outcomes of other components of this external evaluation. For example, a 
number of the teacher comments refer to the value attached to sending in solutions to the 
problems provided on the NRICH website, yet the website evaluation reported in the 
next section suggests that it is not at all clear from the website that solutions are 
welcomed, nor how (or where) these might be sent in to NRICH. A number of teachers 
referred to using NRICH mainly as a source of teaching ideas (and, in one case, the 
teacher admitted to preferring not to tell the class about the source!). Other teachers 
used NRICH problems with a wide range of pupils (one teacher commented, “I use ‘Let 
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me try’ with infants and low ability pupils very successfully.”), or mentioned finding it 
difficult to fit in NRICH in a busy schedule or scheme of work. All these issues surfaced 
in the selected case studies reported in section 6 of this report. 



4.3.5 Summary of the teacher perspective 

The overwhelming majority of teachers who answered the questionnaire were from 
England. US teachers were the next highest category, but at less than 5% of the total. 

Around a third of the all the teachers worked in secondary comprehensive schools. 

About the same proportion were from primary (junior or infant) schools. More than half 
the teachers worked for a local education authority. Around 10% were from private 
schools (over 70% of which were located in England). Most of the teachers worked in a 
suburban school. About half the teachers who worked in inner city schools (29 out of 67) 
worked in primary schools. The primary teachers tended to work in schools of between 
200 and 499 pupils while the secondary teachers were evenly split between those who 
worked in schools of under 1 000 pupils and those who worked in schools of over 1000 
pupils. The overwhelming proportion of teachers worked in mixed schools (both boys 
and girls). 

Many of the teachers (almost 30% of the respondents) had found NRICH through 
browsing the web. Just under 20% had learnt about NRICH from colleagues. Articles in 
journals and magazines, and inservice courses were also major sources. Most of the 
teachers were relatively new to NRICH, having only been accessing the website for six 
months or less. 

The overwhelmingly majority of teachers accessed NRICH from home. Relatively few 
accessed NRICH (for their own use) in their classrooms. The NRICH facility most used 
by teachers was the ‘resource bank’ of problems. Sizeable proportions of teachers 
(around 40%) said that they never used ‘ask NRICH’, or any of the bulletin boards, or 
sent in pupils’ solutions. Those relatively new to NRICH comprised half of the teachers 
who said that they never used these facilities, and almost exactly the same proportion 
who said they accessed these facilities either most days or most weeks. 

Most of the teachers at primary, middle or preparatory schools accessed NRICH once a 
month or less. All the elements of NRICH aimed at such teachers were popular, with the 
mathematical problems having the slight edge. Teachers in secondary and upper schools 
accessed NRICH more often than their primary colleagues. The ‘monthly problems’ and 
the ‘tough nuts’ were especially popular, with around two-thirds of all secondary 
teachers accessing them at least once a month. Approximately half of all secondary 
teachers also checked the news and the articles at least once a month. 

A clear majority of all the teachers (both primary secondary) looked forward to each new 
NRICH issue, with most being aware of the variety of NRICH facilities. While the 
majority said they mainly used NRICH as a source of problems for use in teaching with 
their classes, few said they primarily used NRICH for homework. Almost half said that 
they had encouraged pupils to access NRICH independently of school. Most teachers 
said that they did not have a Maths Club for pupils based around NRICH. 

The teachers recommended NRICH not only to their more able pupils in mathematics but 
more widely. Nevertheless, a majority agreed (many strongly) that NRICH was 
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particularly good for pupils who have a talent for mathematics. Almost 85% of teachers 
agreed or strongly agreed that the NRICH website was well-designed, while only around 
10% claimed to find the website difficult to use. Many teachers thought that NRICH 
made them feel that they could share issues with other mathematics teachers. Few 
teachers thought NRICH would be better as a printed magazine. Only a tiny proportion 
of the teachers who completed the questionnaire said that they were registered with 
NRICH. 

A majority of teachers thought that using NRICH had made their pupils more interested 
in mathematics. About 15% of the teacher respondents added a comment to their 
questionnaire. Virtually every comment was positive and illustrated how important 
NRICH was to many of the teachers who accessed the site, and hence to their pupils. 
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4.4 The Perspective of the ‘ Friends ofNRICH’ 

A total of 28 1 responses were received to the questionnaire addressed to the various 
‘friends ofNRICH’ (those respondents who classified themselves as neither school pupils 
or students, nor school or college teachers). As noted in section 3.3.1, close scrutiny was 
paid to the data in order to sift out, from the data for analysis, incomplete responses, 
unacceptable or frivolous responses, and multiple submissions from the same respondent. 
It was anticipated that a considerable number of incomplete responses to the 
questionnaires would be received. The close inspection of the original data was designed 
to improve the validity of the data set used for analysis and hence the reliability of any 
conclusions drawn from the analysis. The result of this process are given below in Table 
37, showing that 67 responses were accepted for analysis. 



Table 37: Responses to the ‘friends ofNRICH’ questionnaire 



total number of responses to the ‘friends’ questionnaire 


281 


unspecified in every field 


36 


unspecified in almost every field (at the most, three answers specified) 


101 


commented that they had only just found the site 


63 


clearly frivolous respondents (typically only a few responses given and these 
clearly not serious) 


14 


number of ‘friends’ responses accepted for analysis 


67 



The analysis that follows examines the following: 

• who the ‘friends ofNRICH’ are and where they live 

• what the ‘friends ofNRICH’ think ofNRICH and its facilities and how it could be 
improved 



The results are presented below in some detail. A summary is provided in section 4.4.3. 
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4.4.1 Who are the ‘friends of NRICH’? 

Most ‘friends of NRICH’ were parents (Table 38). Just under a quarter were adults 
interested in or studying mathematics. 

Table 38: The types of ‘NRICH friend’ 



a parent 


31 


46.27% 


an adult interested in mathematics 


15 


22.39% 


a lecturer in higher education 


6 


8.96% 


an inspector 


6 


8.96% 


a student teacher 


5 


7.46% 


a school governor 


2 


2.99% 


an adult studying mathematics 


1 


1.49% 


not specified 


1 


1.49% 


Total 


67 


100.00% 



Over half of the ‘friends of NRICH’ lived in England (Table 39). The next largest group, 
although less than 15% of the total, resided in the USA. 



Table 39: Place of domicile of ‘friends of NRICH’ 



England 


40 


59.70% 


USA 


10 


14.93% 


Australia 


4 


5.97% 


Wales 


2 


2.99% 


Denmark 


1 


1.49% 


India 


1 


1.49% 


Isle of man 


1 


1.49% 


New Zealand 


1 


1.49% 


Oman 


1 


1.49% 


Singapore 


1 


1.49% 


not specified 


5 


7.46% 


Total 


67 


100.00% 
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As with the teachers, most ‘friends of NRICH’ had found NRICH from browsing the 
web (Table 40). Colleagues, and articles in journals and magazines, were also important 
ways that they found out about NRICH. 



Table 40: How the ‘friends of NRICH’ found out about the site 



browsing the web 


32 


47.76% 


a colleague 


9 


13.43% 


an article in a journal 


9 


13.43% 


a school teacher 


4 


5.97% 


a family friend 


2 


2.99% 


an INSET course 


2 


2.99% 


another family member 


2 


2.99% 


Gifted and Talented email group 
(Australia) 


2 


2.99% 


a University lecturer 


1 


1.49% 


recommended by National Association 
of Gifted Children 


1 


1.49% 


reviewing NRICH for a professional 
teaching journal 


1 


1.49% 


not specified 


2 


2.99% 


Total 


67 


100.00% 



Most of the ‘friends’ were relatively new to NRICH (Table 41). Almost 78% had been 
accessing NRICH for she months or less. 



Table 41: How long the ‘friends of NRICH’ had been accessing the site 



a month 


33 


49.3% 


six months 


19 


28.4% 


a year 


9 


13.4% 


a year or two 


1 


1.5% 


not specified 


5 


7.5% 


Total 


67 


100% 
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Two thirds of NRICH’ s ‘friends’ access the site from home; one third from their place of 
work (Table 42). 



Table 42: Where the ‘friends of NRICH’ accessed the site from 



home 


44 


65.67% 


work 


21 


31.34% 


home and work 


1 


1 .49% 


University 


1 


1 .49% 


Total 


67 


100.00% 



4.4.2 What the ‘friends of NRICH’ think of NRICH 

All but one of the ‘friends of NRICH’ respondents indicated on the questionnaire how 
they made use of NRICH. Almost all the parents said that they used NRICH as a source 
of interesting mathematics to do with their own children. Four of the parents (two from 
England and two from the USA) said that they were educating their children at home, 
and two others said that they worked for an advice service or group for the home 
educated. 

Amongst the comments from parents were the following: 

“I look for interesting math problems for my 8 year old son. Not that he is a 
genius, but the usual additions and subtractions -which he gets as homework 
bore him. ” 

Parent from Oman. 

“As educational fun for my 10 year daughter”. 

Parent from England. 

“Me and my 10 year old son have a monthly contest as to who can answer the 
most questions correctly!" 

Parent from England. 

The adults interested in or studying mathematics used NRICH as a source of interesting 
problems. The comment below is typical of this category of ‘friend if NRICH’: 

“I enjoy testing my very rusty knowledge on the problems”. 

An adult from England interested in mathematics. 

There were also other uses made of NRICH. One adult was using the website as a source 
of mathematical ideas for their daughter who was just starting out as a newly qualified 
teacher in England. Other categories of ‘friends of NRICH’, such as University lecturers 
and local authority inspectors, were also using the site as a source of mathematical ideas. 

More than 80% of the ‘friends of NRICH’ made some comment about what they liked 
about NRICH. The most commonly complimented aspects of NRICH, with comments 
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coining from all categories of ‘friend’, related to the range of activities, the presentation 
and the interaction facility. Below are some typical comments: 

“Interesting problems, attractively presented, with variety each month. Good 
range of skills required in each set. ” 

Parent from England. 

“The problems are clear, thought provoking and fun. ” 

Parent from England. 

“It shows how maths can be presented in a positive, enjoyable and effective 
way”. 

School governor from England. 

“The questions are stimulating and enjoyable and the range of difficulty 
admirable. The site layout is clear and easy to navigate, and pages load 
quickly. ” 

Parent from England. 

“The site is visually attractive, well organised, navigation seems 
straightforward The breakdown amongst areas seems sensible. I like it very 
much that children can get into direct discussions with University students. ” 

An adult interested in mathematics from England. 

Several of the adults interested in mathematics mentioned the personal growth aspect. 
Below are examples of this type of comment: 

“I enjoy the clear solutions for the mid-level problems and the articles. ” 

An adult interested in mathematics from the USA. 

“It fuels my interests in Mathematics and spurs me on to continue learning new 
methods of solving problems. ” 

An adult interested in mathematics from Singapore. 

Only one ‘friend of NRICH’ respondent mentioned the more able pupil in mathematics: 

“It is suitable for gifted children. My daughter attends school and is 
significantly under-challenged, so she benefits by using it [NRICH] at home. ” 
Parent from England. 

Less than half of the ‘friends of NRICH’ chose to comment on ways in which they 
thought NRICH could be improved and a number of these were to the effect that they 
could not think of anything or that they had not been accessing NRICH long enough to 
properly suggest an improvement. The suggestions that were made included the 
following (in no special order): 

• enrichment in science and English language 

• having the problems translated into other languages 

• a zipped (compressed) archive containing each month's pages, text and graphics, so 
these could be downloaded in one go and browsed later off-line 

• information on new developments in Mathematics, eg solution found for Fermat's Last 
Theorem, etc 
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• regular e-mail message to users to inform them of a new edition, how harder puzzles 
are going, etc 



4.4.3 Summary of the perspective of the ‘friends of NRICH’ 

Most of the people who completed the ‘friends of NRICH’ questionnaire were parents. 
Just under a quarter were adults interested in or studying mathematics. Over half of the 
‘friends of NRICH’ lived in England. The next largest group, although less than 15% of 
the total, resided in the USA. As with the teachers, most had found NRICH from 
browsing the web. Colleagues, and articles in journals and magazines, were also 
important sources of knowledge. Most of the ‘friends’ were relatively new to NRICH. 
Over three quarters had been accessing NRICH for six months or less. Two thirds of 
NRICH’ s ‘friends’ accessed the site from their home; one third from their place of work. 

Almost all the parents said that they used NRICH as a source of interesting mathematics 
to do with their own children. Four of the parents (out of 31) said that they were 
educating their children at home, and two others said that they worked for an advice 
service or group for the home educated. Other categories of ‘friends of NRICH’, such as 
adults interested in mathematics, University lecturers and local authority inspectors, used 
NRICH as a source of mathematical ideas. 

When asked what they liked about NRICH, all categories of ‘friend’ commented on the 
range of activities, the presentation and the interaction facility. The NRICH site was 
complimented as providing interesting problems and being attractively presented and 
easy to navigate. Only one ‘friend of NRICH’, a parent, specifically mentioned the 
suitability of NRICH for the more able pupil in mathematics. The ‘friends’ made a 
number of helpful suggestions for improving the NRICH service that might be profitably 
considered by the NRICH team. 
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4.5 Registrations with NRICH 

Registration with NRICH is a straightforward process and is open to all. It brings 
additional benefits associated with the NRICH bulletin board system, an electronic 
message database where NRICH users can read or send messages. These messages are 
archived so that earlier messages can be read. Some parts of the NRICH bulletin board 
system are open to all. For some parts, users need to register. Straightforward 
instructions on how to register online at the NRICH website are provided at the site. 

The NRICH bulletin board system has the following public areas available to any user of 
the NRICH website: 

• Announcements: where the NRICH team post announcements of general interest. 

• The Statue of Anonymous 10 : a ‘moderated’ facility for reading or joining in mathematical 
discussions. New topics for discussion can also be posted. All contributions are read by a 
member of the NRICH team before being posted on the bulletin board. 

NRICH also provides several ‘closed’ (or semi-closed) discussion areas. In order to 
make full use of these facilities, NRICH users have to register. These discussion areas 
include: 

• One-to-one: pupils wanting to discuss a problem can post a message. NRICH has a team of 
students studying mathematics at university who monitor this particular list and send a 
suitable reply. Only registered pupils can use this facility. 

• Open discussion: anyone can read these discussions. Only registered users can post messages. 

• NRICHtalk: only open to registered users. 

• TeacherTalk: only open to registered teachers. 

• TechTalk: similar to the Open discussions, anyone can read these messages. Anyone can post 
a message too, but those from unregistered users are vetted. 

The NRICH project maintains a database of users that have registered with NRICH. A 
table of the cumulative totals of registrations are given below (Table 43). This data is 
also presented graphically in Figure 1 below. The data was provided by the NRICH 
project. 

Table 43 shows steady a growth in registrations over the period October 1997 to 
October 1999. By October 1999, nearly 1400 pupils and just over that number of 
teachers had registered with NRICH. 



10 Named after a famous statue in Budapest where a group of school children with a serious interest in 
mathematics used to meet A number of these children went on to become leading mathematicians of 
the 20th century, including Erdos, Turan, Tibor, and Szekeres. 
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Table 43: Registrations with NRICH (data supplied by NRICH) 



Date/Month 




Pupils 


Pupils 

cumulative 




Teachers 


Teachers 

cumulative 




Mature 


Mature 

cumulative 


Oct-97 




42 


42 




26 1 


26 








Nov-97 




39 


81 




96 


122 








Dec-97 




37 


118 




46 


168 








Jan-98 




22 


140 




41 


209 








Feb-98 




17 


157 




41 


250 








Mar-99 




21 


178 




68 


318 








Apr-98 




52 


230 




38 


356 








May-98 




27 


257 




50 


406 








Jun-98 




21, 


278 




29 


435 








Jul-98 




43 


321 




23 


458 








Aug-98 




7 


328 




8 


466 








Sep-98 




19 


347 




24 


490 








Oct-98 




123 


470 




95 


585 








Nov-98 




71 


541 




63 


648 








Dec-98 




38 


579 




46 


694 








Jan-99 




81 


660 




87 


781 








Feb-99 




60 


720 




54 


835 








Mar-99 




129 


849 




81 


916 








Apr-99 




34 


883 




52 


968 




7 


7 


May-99 




59 


942 




56 


1024 




15 


22 


Jun-99 




78 


1020 




55 


| 1079 




17 


39 


Jul-99 




93 


1113 




71 


1150 




15 


54 


Aug-99 




48 


1161 




49 


1199 




16 


70 


Sep-99 




92 


1253 




126 


1325 




32 


102 


Oct-99 




103 


1356 




113 


1438 




36 


138 




Figure 1: Cumulative NRICH registrations 
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4.6 Summary of the Findings from the Questionnaire Data 

Analysis of the questionnaires revealed that most NRICH users (including pupils, 
teachers, and ‘friends of NRICH’, such as parents) lived in England. This was 
particularly true of teachers. Over two-thirds of all the pupils who accessed NRICH were 
boys. Approximately the same proportion was white. The next largest ethnic group was 
Chinese pupils. About half of all pupils accessed NRICH at school, usually in an IT 
room, with almost exactly the same proportion of the pupils accessing NRICH at home. 
There was no difference in this pattern of access between boys and girls. 

The majority of teachers who accessed the NRICH site worked in the state sector; 10% 
were from private schools. Under half of the teachers worked in the primary or middle 
schools, with a slightly larger proportion from secondary schools (state and private). The 
majority of all NRICH users were relatively new to NRICH, having been accessing the 
website for six months or less; most accessed NRICH from home. The most frequent use 
of NRICH was as a source of interesting mathematical problems. 

Nearly 1400 pupils, and just over this number of teachers, had registered with NRICH by 
October 1999. Most users of the NRICH site were not registered with NRICH. Only 
some of the registered users who accessed the site made much use ofthe bulletin board 
facilities. The NRICH site was complimented by all categories of user as providing 
interesting problems, being attractively presented and generally easy to navigate. 
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5. Website Evaluation 



5.1 Introduction 

This element of the evaluation involved submitting the NRICH site to critical review and 
comparing it to two other selected websites that also provide regular mathematical 
problems and/or some form of answering service. In addition, data from the log of 
accesses to the NRICH website are used to examine some aspects of the use of the 
NRICH site during the period September 1998 to August 1999. 



5.2 A Critical Review of the NRICH Website 

The website evaluation criteria used by the evaluation team to review the NRICH site 
were adapted from those described by Branch et al (1999), Coe and Land (1998), 
Shneiderman (1997), and by Testa (1998). The criteria cover the following: 

• Authority 

• Accuracy 

• Currency 

• Navigation and Design 

• Applicability and Content 

• Scope 

• Audience Level 

• Quality 

• Awards 

Associated with each of these factors is a list of questions that serve to focus the 
evaluative judgement. The full evaluative criteria are given in Appendix C. 

A new design for the NRICH website was launched in July 1999. This critical review 
was undertaken both before and after the launch of the new design. Only the results of 
the review of the new design are presented below as the new design was a marked 
improvement on the old design and, now that the old design no longer exists, information 
on the old design is unnecessary. For each criteria, the NRICH website was examined 
using the questions given in Appendix C and a judgement made as to the rating of the 
NRICH site on a four point scale (unsatisfactory, satisfactory, good, high) 

Authority: the NRICH website was judged to have a high authority rating. The source of 
the information is given, there is a statement of the aims and objectives of the site, the 
authors of the materials can be contacted and represent an established academic 
institution. 

Accuracy : the NRICH website was judged to have a high accuracy rating. The material 
is from a reliable source. There is no advertising that might conflict with the aims of the 
site. 
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Currency: the NRICH website was judged to have a high currency rating. Material is 
updated monthly. The date of updating is clearly visible. Archives of past materials are 
well-maintained. 

Navigation and Design : the NRICH website was judged to have a high rating for 
navigation and design. The resource is organised into manageable chunks of information 
that can be browsed easily. There is a contents column or index that describes what is 
contained within the site. There are good navigational links within the pages. Links are 
clearly labelled and images are used that support the navigation process. A search facility 
is available for both mathematical topic and keyword. Both online help and interactive 
help (for example, through e-mail contacts) are provided. Pages in a form suitable for 
printing are available when appropriate. There is some difference in design, feel, and 
navigation between the part of the site referred to as ‘NRICHprimary’ and the rest of the 
site which has the potential to interfere with easy navigation of the site. 

Applicability and Content : the NRICH website was judged to have a high rating for 
applicability and content. The material is relevant to the intended audience and there is 
good coverage of the relevant content with suitable variation in how the content is 
presented. 

Scope: the NRICH website was judged to have a high rating for scope. The purpose of 
what is included on the NRICH site is clear and matches the mission statement of the 
project. 

Audience Level: the NRICH website was judged to have a good rating for audience 
level. Much of the site is clearly aimed at pupils, yet the part of the site referred to as 
‘NRICHprimary’ has pages called ‘teacher’s notes’. 

Quality: the NRICH website was judged to have a high rating for quality. The material is 
well-written with complex ideas introduced and discussed with clarity. Good use is made 
of illustrations and diagrams. 

Awards: the NRICH website makes no mention of awards but then there are issues of 
authenticity associated with website awards. There may be as many as 800 possible 
different ‘awards’ that a website can be granted. There are no guarantees about the 
credibility of such awards. 



5.3 Comparing the NRICH Provision 

There are many websites that provide mathematical puzzles, games and problems. The 
NRICH site itself for example, lists 40 sites world-wide.in its section of links to other 
mathematics sites. Not all these particular sites provides a comparable service to NRICH. 
One website, the (US) Math Forum, funded by the US National Science Foundation, has 
features called ‘Problems of the Week’ and ‘Ask Dr. Math’. 

The Math Forum's ‘Problems of the Week’ can be found at the following web address: 
http://forum.swarthmore.edu/pow/ The ‘Problems of the Week’ are designed to provide 
“creative, non-routine challenges for students in grades three through twelve. Problem- 
solving and mathematical communication are key elements of every problem.” Separate 
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problems are provided for elementary school pupils, middle school pupils, and, for high 
school students, on geometry, algebra, discrete mathematics, and trigonometry and 
calculus. Solutions can be submitted and the website includes an archive of past problems 
and student solutions (categorised and searchable). The Math Forum claims that during 
the 1998-99 school year, more than 18,500 students submitted solutions to the 
‘Problems of the Week’ with over 4,500 schools participating. 

‘Ask Dr. Math’ is an answering service in which ‘Dr. Math’ replies to questions from 
pupils and their teachers about school mathematics. Questions can be about homework, 
puzzles, mathematics contest problems, or any other mathematical topic. The archive of 
past questions and answers is categorised and searchable. Categorisation is by school 
level then mathematical topic. The Math Forum claims that over 5,000 questions are 
already answered. The ‘Ask Dr. Math’ service does not promise to answer every 
question posed by pupils or their teachers. 

A UK-based mathematics answering service is provided by ‘Maths Help’. The aim of this 
website is to provide “free help and advice with problems in Mathematics and Statistics 
at GCSE, A-level, BTEC, GNVQ and Foundation year degree level”. The website for 
‘Maths Help’ can be found at the following web address: 
http://www.maths-help.co.uk/index.html 

The site claims to be run “by a partnership of experienced mathematics teachers who 
believe that the Internet has huge potential as a source of information and a means of 
communication in the field of education”. They say that “other websites which offer help 
with maths tend to be based in universities, and often use university students to reply to 
the queries”. They claim that using people who are not qualified teachers to reply to the 
problems can mean that the answers are “sometimes too technical and could go over the 
heads of the reader”. Maths Help is available round the clock and aims to provide a 
response by e-mail within 24-48 hours. The professed mission of ‘Maths Help’ is to 
become “the website of choice for UK students of mathematics at upper secondary and 
tertiary level by September 2000”. 

Elements of the Math Forum’s ‘Problems of the Week’ and ‘Ask Dr. Math’ were judged 
to be good or high on many of the website evaluation criteria. NRICH scored better in 
terms of navigation and design, and quality of the provision. The UK-based answering 
service ‘Maths Help’ scored poorly in terms of authority. It is unclear from the site who 
the authors are - only an anonymous ‘webmaster’ e-mail address is provided. 



5.4 NRICH Server Statistics 



Records of the ‘hits’ (in terms of requests for pages) on the NRICH website are archived 

by the NRICH server, as is other data (such as where visitors are accessing the site from, 

the files accessed or downloaded, errors and alerts, etc). The data are available on the 

NRICH website at the following pages: 

http://nrich.maths.org/stats98.html 

http ://nrich. maths, org/ stats-to-6Sep99 .html 

In this section, only the number of ‘hits’ on the NRICH server are considered. Even so, 
as mentioned in section 3.3.2, server log files are far from straightforward to interpret 
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(Bertot etal 1997, Buchanan and Lukaszewski 1997, Stout 1997). For example, many 
internet providers route their internet traffic through ‘proxy’ or ‘cache’ servers so that 
the NRICH server logs do not easily or accurately trace the number of users, but rather 
they reflect the number of accesses by the referring servers. As the number of ‘cache’ 
servers is continuing to increase, comparing statistics, even over the time scale of a year, 
may well not be reliable. 

Table 44 below shows the total ‘hits’ for each month for the period September 1998 to 
August 1999. The same data is also presented in a line graph (Figure 2). 

Table 44: Accesses to the NRICH site by month from September 1998 to August 1999 



Month 


NRICH website hits 


Sep-98 


16364 


Oct-98 


19074 


Nov-98 


34347 


Dec-98 


33547 


Jan-99 


25939 


Feb-99 


47648 


Mar-99 


76210 


Apr-99 


68387 


May-99 


87184 


Jun-99 


65599 


Jul-99 


82279 


Aug-99 


44661 




Figure 2: Accesses to the NRICH site by month from September 1998 to August 1999 

The NRICH server statistics show that ‘hits’ on the NRICH site have risen from around 
25000 a month to more than 60000 a month over the year from September 1998 to 
August 1999. This is likely to be the result of more people accessing the NRICH site 
more often. An unquantifiable element is also no doubt due to the increase in web 
‘browsing’ as internet access becomes more common. The results from the 
questionnaires (section 4) indicate that most of those accessing the NRICH site to make 
use of the materials had been doing so for a relatively short time, with ‘browsing the 
web’ being the most common way of finding the site. The large number of incomplete 
questionnaire returns, which had to be eliminated from the analysis, may be a further 
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indication that a proportion of the ‘hits’ on the NRICH site may be from browsers just 
happening on the site. As indicated above, though, the increasing use of ‘cache’ servers 
could just as well mean that the server logs underestimate the ‘hits’ on the site as 
multiple uses accessing the site through the same ‘cache’ server do not show as separate 
‘hits’. 

At the time of writing, a new format of server log record has been made available on the 
NRICH site. This new format provides greater detail on the accesses to the NRICH site 
but uses a slightly different method of recording the ‘hits’ data. The result is slightly 
different data to that presented above. This is another example of how web site server 
statistics need to be treated with some caution. The new NRICH server statistics are at 
http .7/ nrich. maths . org/logs/ 

5.5 Summary 

The NRICH website was judged by the evaluation team to score highly on each of the 
website evaluation criteria. The new design of the site, launched in July 1999, was judged 
to be attractive, functional, easy to navigate, and contain high-quality materials. NRICH 
compared very favourably with other sites that provide mathematical puzzles, games and 
problems, and/or an answering service. The NRICH server statistics show an increase in 
accesses to the site which is likely to be the result of more people accessing the NRICH 
site more often. 
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6. Selected Short Case Studies 



6.1 Introduction 

This element of the evaluation provides additional evidence of the uptake, use, and ICT 
provision in schools that access NRICH. The evidence is in the form of three short case 
studies of schools where at least one teacher had been accessing the NRICH website for 
at least six months. Every effort was made to obtain a stratified random sample of such 
schools (the stratification being primaiy, secondary, and private school). However, for 
each stratification, the population from which the sample had to be chosen was too small 
for the sample to be considered properly random. Section 3.2.3 of this report describes in 
more detail how the selection of schools was made. 

Five schools were visited, two primary, two secondary, and one private school, all in 
England. The schools were located in different types of area: inner-city, urban, suburban, 
and rural. Data were collected in the form of audio-taped interviews with teachers and 
pupils where possible, classroom observation notes where possible, and the records of e- 
mail conversations. Interview and observation schedules are given in Appendix D. 

It did not prove possible to collect the same amount and quality of data in all five 
schools. Accordingly, short case studies are presented below of three schools, one of a 
primary school, one of a secondary school, and one of a private school. In each case, a 
draft of the case study description was checked by the teacher at the particular school for 
factual accuracy and to ensure that neither the individual school nor any individual 
teacher or pupil could be identified, the latter as part of the guarantee of anonymity for 
all those interviewed and observed. 



6.2 Case study 1: inner-city primary school 

This average-sized inner-city mixed primary school is surrounded by mostly 19 th centuiy 
housing in a densely populated area just south of the centre of a major UK conurbation. 
The socio-cultural and ethnic backgrounds of the pupils are veiy varied. A significant 
proportion of the pupils come from less affluent homes and around half are from black or 
Asian ethnic groups. Many of the pupils are eligible for free school meals. A significant 
number of pupils in each year group have special educational needs. 

Although not a pilot school for the UK national numeracy project, the school began 
adopting the approach specified in the National Numeracy Strategy (NNS) about a year 
before this was required to be introduced. Interviews with the curriculum co-ordinator 
for mathematics and classroom observations showed that staff at the school employ a 
good range of teaching methods based around the three-part lesson stipulated in the 
NNS. Pupils are enthusiastic and well motivated. The results of national assessments in 
mathematics show that pupil attainment is close to being in line with the national 
average. The curriculum co-ordinator for mathematics works hard to support staff 
through advice and professional development opportunities. Lessons are carefully 
planned and make frequent and effective use of resources, including the use of 
information technology when appropriate. Every class has a computer with a basic 
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package of software. A newly developed ICT area is beginning to provide more 
sophisticated ICT opportunities for pupils including access to the internet. Teachers are 
generally confident about using ICT and pupils at all ages are keen and interested. 

The school mostly uses NRICH as a source of ideas for extending its more able pupils in 
mathematics. The school’s local education authority has encouraged its schools to 
develop provision for the more able for some time and the school is aware of the 
Government’s ‘Excellence in Cities’ initiative (DfEE 1999) with its “gifted and talented 
children strand”. The curriculum co-ordinator provides printed out copies of what she 
considers to be suitable problems for the various classes and individual class teachers 
choose from these as they see fit. Most of the teachers use the NRICH problems 
occasionally to ‘stretch’ the more able in mathematics when they have finished their 
regular classwork. A few NRICH problems have been incorporated into the school’s 
scheme of work for mathematics. Some of the mathematical problems have also been the 
source of “off the cuff” surprise lessons which have enlivened the routines of both the 
teachers and their pupils. Occasionally a class has really taken to a particular problem and 
on one occasion a solution was sent into NRICH. The pupils in that particular class were 
very excited to see their solution published on the NRICH site. 

With the school now connected to the internet, some pupils have accessed the NRICH 
site ‘online’ and this is seen as a good new facility. The curriculum co-ordinator for 
mathematics is concerned that software provision in mathematics in the school is not as 
good as she would like and the advantage of accessing NRICH is that the pupils are 
using a computer to do mathematics but without the cost associated with purchasing 
software. She would like to see the NRICH site being more ‘interactive’, and hence 
appearing to the pupil user to be more like a piece of conventional software, but 
appreciates that this might not be technically possible. The school has also been involved 
with an internet initiative sponsored by a major UK supermarket chain which made it 
relatively straightforward for the school to publish material about its locality on the 
world wide web. The curriculum co-ordinator wonders whether it would be possible for 
NRICH to develop the facility where pupils (or schools) could publish their solutions on 
the world wide web for other pupils (or schools) to see 11 . 

Although the curriculum co-ordinator is registered with NRICH she makes little use of 
the e-mail facilities, mostly through lack of time. None of the pupils at the school are 
registered as yet, primarily because use of NRICH, particularly online, is still something 
relatively new and online usage has associated costs and is not especially fast (the school 
does not have an ISDN line 12 , for instance). The staff are currently somewhat wary of 
internet use by pupils that is not closely supervised. Parental permission is beginning to 
be routinely sought for all internet usage by pupils. The staff are also mindful of concerns 
about pupils of primary age corresponding with strangers by e-mail even when the 



11 These opportunities are beginning in other school subjects. The Research and Graduate School of 
Education at the University of Southampton is home to Scl-Joumal, an award-winning on-line 
publication for science students giving them the opportunity to publish work done in their school or 
college science classes so that other science students around the world can read it The web address of 
Scl-Joumal is http://www.soton.ac.uk/~plf/ScI-Joumal/index.html 

12 Integrated Services Digital Network: an ISDN telephone line is digital and can cany far more 
information than a standard (analogue) UK telephone line. An internet connection via an ISDN line 
can appear to be up to four times faster than a connection through a standard (analogue) telephone 
line. Currently, an ISDN line costs far more to install and rent than a standard telephone line. 
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authenticity of those ‘strangers’ is not really in doubt. As the staff become more 
experienced and more confident with using the internet they think that they will begin to 
use e-mail more. 

Despite the curriculum co-ordinator’s comments on limited usage of NRICH, she does 
think that it has had some impact on the teaching and learning in the school. Class 
teachers look forward to seeing new interesting problems and the more able pupils have 
enjoyed the challenges and the opportunities, when these have been possible to arrange, 
to discuss their strategies and solutions with other children in the school of a similar level 
of attainment. Staff have also got together, on occasion, to discuss various NRICH 
problems and how children go about solving them. The curriculum co-ordinator would 
like to build on the use the school makes of NRICH but, she says, this has to fit with the 
school’s developing implementation of the National Numeracy Strategy (NNS). She has 
some concerns that the NNS means less opportunity for fitting in the sort of problems 
available through NRICH. She does have a clear picture of the developments she would 
like to make, but, she says she would like a little respite from Government initiatives and 
a period of stability in which to reflect further. 



6.3 Case study 2: suburban secondary school 

While this larger-than-average-sized 11-16 age-group mixed secondaiy school is located 
on the outskirts of a large village, it serves a catchment area that includes the commuter 
belt of a medium sized English city and a nearby large town. In effect it could be 
classified as a suburban school. The proportion of higher social class households in the 
area served by the school is above the national norm. There are few pupils at the school 
eligible for free school meals. Most of the pupils in the school are white, with a very few 
from other ethnic groups. There are few pupils with statemented special needs. 

Mathematics is taught by a well-qualified, hardworking team in a specialist suite of 
classrooms. Classroom displays of pupils’ work and of aspects of mathematics and the 
work of mathematicians are of a very high standard. The department runs two extra- 
curricular mathematics clubs for pupils each week, one with a varied programme open to 
all pupils, one that focuses more on mathematical games and is aimed primarily at the 
less able pupil. Teaching quality is very high and planning at both classroom and 
departmental level is very thorough. Classes are set by attainment at entry to the school 
at age 1 1 and there is a detailed scheme of work for each of three attainment groups: the 
most able 10-20%, the intermediate level (most pupils), and the foundation level (the 
least able 10% or so). The results of the national attainment tests in mathematics at the 
end of Key Stage 3 (pupils aged 14) show performance is above the national average. At 
age 16, pupil mathematics attainment is generally in line with, and sometimes above, the 
national average. The higher attaining pupils have the opportunity to study for and enter 
the separate GCSE qualification in Statistics. Most of the school's pupils go on to further 
study at nearby 16-19 colleges. 

Provision for information technology in the school is good overall, the school having 
recently acquired an impressive new ICT suite in a new specialist extension. The school 
is currently bidding to the Department for Education and Employment to become a 
specialist technology school (an annual competition bringing substantial additional 
funding). The suite of mathematics classrooms is some distance from the ICT suite and 
the IT facilities within the mathematics area are somewhat dated with the machines 
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available for pupil use not capable of internet access. The single computer located in the 
mathematics department office for staff use is more up-to-date but the mathematics area 
is not networked so internet access is not possible from there. The mathematics 
department makes good use of graphing calculators and a graph-plotting package 
available on the computers in the mathematics suite. Timetable pressure on the school’s 
ICT suite is fairly intense which restricts spontaneous usage. The mathematics 
department’s usage of the suite is quite innovative with a mathematics resources 
intranet 13 having been developed by staff giving pupils opportunity to search for 
information relating to various topics in mathematics. 

The school’s local education authority has encouraged its schools to develop provision 
for the more able for some time and the school has an ‘able pupil policy’ that targets the 
top 10-20% of each year group. The mathematics team is aware that the school has a 
few exceptionally able pupils in mathematics. While these pupils, themselves, do not wish 
to be publicly identified as ‘exceptionally able’, the department provides extension 
material as part of their top level scheme of work with the aim of broadening and 
deepening the understanding these pupils have of mathematics. 

Mathematical problems taken from the NRICH website constitute just one source of 
material that contributes to that available to teachers to supplement the regular scheme 
of work. Other resources, both printed and internet-based, are also used as sources of 
supplementary extension material. One of the mathematics teachers in the department 
prints off what he considers to be suitable short problems or puzzles from the NRICH 
website, usually from his home, and these are shared with the other teachers in the 
department. This is usually done weekly and often generates enthusiastic and animated 
discussion at break time in the mathematics department office, particularly when an item 
presents a new way of approaching a standard topic. Some NRICH items have been 
incorporated into the various mathematics schemes of work, others are just added to 
extensive resource banks that teachers can draw on as they see fit. The more challenging 
problems and the ‘tough nuts’ are generally seen as things that take time away from an 
already crowded curriculum that already contains the right amount of such material. In 
general, NRICH is seen (and used) as a source of useful problems, not something that 
replaces what the mathematics department does in terms of ‘publishing’ pupil solutions, 
providing assistance to pupils, or running a mathematics ‘club’. The department values 
NRICH greatly as a source of interesting problems, but only as one source amongst the 
many that the department uses to enliven its mathematics teaching and the mathematical 
experience of its pupils. 

The teacher who prints off the NRICH materials is registered with NRICH but has found 
little need to access those additional NRICH facilities available to those who choose to 
register. On occasion, the NRICH site has been mentioned to interested parents at 
parental evenings and it is thought likely that some of them would have accessed the site. 
To date, and for the reasons outlined above, pupil usage of NRICH in the school is not 
extensive. The department does have well-developed plans to improve its own ICT 
resources, in which case more usage of NRICH may result. Such developments, 
however, depend on the availability of sufficient funding, and it is not yet clear to the 
department when such funds are likely to become available. 



13 An ‘intranet’ provides s imil ar services -within an organisation to those provided by the internet outside 
it, but without necessarily being connected to the internet 
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6.4 Case study 3: small-town private school 

This medium-sized girls 11-18 private (fee-paying) school takes both day and boarding 
pupils and is located in an attractive, small market town. While a limited number of 
scholarships are available, most pupils come from families well able to afford the fees 
that the school charges. Entry to the school is competitive with examinations taking 
place in the January prior to entry to the school in the following September. Most of the 
pupils in the school are white, with a very few being from other ethnic groups. The 
school emphasises academic achievement in an atmosphere that is generally relaxed. 
Relationships between pupils and staff are polite but not overly formal. 

Mathematics is taught by a well-qualified team of teachers in general classrooms. The 
school is not able to provide specialist mathematics classrooms and so the mathematics 
facilities within the classrooms are limited. None of the general classrooms have very 
much storage space nor are themselves equipped with computers. Mathematics teaching 
is relatively formal and focuses on techniques, skills and developing mathematical 
thinking. Most teaching is based on textbooks, with some practical work and use of ICT 
(both in the form of computers and graphic calculators) where this is seen to be useful. 
Over 90% of the pupils gain a grade C or above in GCSE mathematics at the age of 16, a 
much higher percentage than the national norm. The number of pupils opting to study 
mathematics at advanced level (A-level) in the sixth form (16-18 year olds) has increased 
in recent years, with up to 20% of students in the cohort taking the subject. The ‘Further 
Mathematics’ A-level only operates when there is demand and this does not happen 
every year. Each year around 95% of the students go on to University. In most recent 
years no more than one or two students have chosen to study mathematics or a closely 
related subject. 

ICT facilities in the school have improved in the last five years or so with the 
development of an ICT room with internet access and the provision of a computer in the 
staffroom also with internet access. Staff and pupils are able to have their own school e- 
mail addresses. 

Neither the school nor the mathematics department make specific additional provision 
for the exceptionally able, in mathematics or any other subject, above and beyond the 
specialist teaching provided for the top setted classes. Early entry for mathematics GCSE 
(at age 15 rather than 16) is being tried for the most able, with successful pupils likely to 
begin some A-level modules in the following year. The mathematics department runs a 
lunchtime mathematics ‘club’ for year 7 pupils (1 1-12 year olds) with a variety of 
activities such as mathematical models, games and codes. An annual mathematics event 
is organised for students in the sixth form (16-18 year olds) in conjunction with the 
private school for boys in the town. 

One use made of NRICH is as a source of mathematical problems. The ‘monthly 
problems’ are displayed on a prominent noticeboard in the school, along with other news 
of a mathematical nature. As well as this, a ‘Puzzle of the Week’ of a more ‘soundbite’ 
nature is posted on the noticeboard (‘Penta Problems’ have been used as a source for 
this). Pupils are encouraged to submit a solution to the mathematics department with the 
attraction of a small prize. The mathematics department is trying a lunchtime ‘maths 
club’ for year 8 and 9 pupils (12-14 year olds) based around the NRICH problems, but 
only a small number attend, partly because of pressure of time in the lunch hour. 
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Pupils at the school are aware of the NRICH site and some have accessed it, mostly at 
school during breaktime or lunchtime access to the schools’ ICT room. A few have 
accessed NRICH from home, but only once or twice. Pressure on the ICT resources 
within the school, and the large range of competing activities, means that pupil’s own 
access to NRICH has been infrequent. The site is generally liked by pupils but for them it 
appears as only one computer usage out of many. Making the website more interactive 
so that problems could be solved online was one suggestion made that would improve 
the site in the eyes of the pupils. A few pupils have sent in solutions with one being 
published by NRICH the following month. Some pupils have registered with NRICH and 
one has made use of ‘askNRICH’ as a way of getting help on a particular problem on 
one occasion. She received a helpful reply about three days later. 

Students studying mathematics in the sixth form (16-18 year olds) have been introduced, 
by one of the teachers, to the PASSMaths website, tin online mathematics magazine 
aimed at a slightly older audience than NRICH that is now also a component of the 
Millennium Mathematics Project. These students found the site interesting but had not 
found occasion to return to it. They did make use of the internet in some of their studies, 
mainly as a source of information when they were asked to do research for one of their 
subjects. Examples of such internet access were given for science, geography and 
history. None of the students said they were ever asked to do such research foF A-level 
mathematics as it was not that type of subject. 

The teacher who accesses the NRICH site is registered with NRICH but rarely if ever 
makes use of any of the wider NRICH facilities. Of the e-mail messages that she receives 
from the NRICH bulletin boards, she finds she has not got enough time to read them all. 
She has not, as yet, made any contribution herself. She has only on rare occasions sent in 
pupil solutions to the NRICH problems. 

The biggest impact of NRICH at the school has been on the staff. Puzzles have always 
formed part of the mathematics curriculum at the school, but the NRICH problems and 
puzzles have provided the staff with new and interesting ideas. That NRICH publishes 
monthly, in colour, and on the internet where it is accessible (and, unlike a magazine, 
does not get buried under other paperwork or in the boot of the car), is a further big 
attraction. Direct impact on pupils is judged to be more limited. The posting of weekly 
puzzles on the noticeboard has successfully raised the profile of mathematics in the 
school and some pupils are beginning to respond. The staff were pleased with what they 
have been able to achieve in the relatively short time (around 6 months) that they had 
been using NRICH and are grateful to the NRICH project for being a vital source of 
mathematical problems and puzzles for them. 



6.5 Summary of the selected short case studies 

Two of the teachers who were the main people interviewed for these three case studies 
had been accessing the NRICH website for about six months; the other teacher for a year 
or two. One had found NRICH from an article in a journal, one had been told about the 
NRICH website by a professional colleague not in the same school, and one had found 
the site while browsing the web. All three teachers were registered with NRICH but had 
made little, if any, use of the wider NRICH facilities. 
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All the three teachers made regular and often frequent use of the NRICH website. In two 
cases NRICH materials were primarily accessed as a source of ideas to support the more 
able in mathematics (in one case as a major source, in the other as just one source out of 
many). In the third case, NRICH materials were used mainly to raise the profile of 
mathematics but this was in a school where most of the pupils were reasonably able in 
any case. In each school the NRICH materials had an impact beyond their usage with the 
most able as teachers discussed the problems and, when suitable, integrated the problems 
into their more general teaching. 

Pupil usage of NRICH in the case study schools was much more varied. Only a few 
pupils in each school would be aware of NRICH and have accessed the site themselves. 
There was evidence of impact on some of the more able pupils, particularly in the two 
schools where special resources for such pupils were not in abundance. The impact was 
in terms of helping these particular pupils to gain a wider appreciation of mathematics 
and raising the profile of mathematics as a subject they could pursue either within school 
or outside school. None of the teachers were able to quantify the impact but all praised 
NRICH as a very valuable resource. 

Overall, the evidence from the case studies suggests that to date, in schools where 
NRICH material is used, it has its biggest impact on teachers. It does this through 
regularly providing novel and interesting problems that often afford a~new way of 
approaching a standard school mathematics topic. 
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7. Selected Case Profiles 



7.1 Introduction 

This fourth element of the evaluation was planned to be in the form of reasonably indepth 
accounts of the experiences of a small sample of pupils in accessing the NRICH website, 
particularly their experience in using the ‘one-to-one’ facility where they can talk over a 
mathematical problem with someone studying mathematics at university (one of the 
distinctive features of the NRICH website). As explained in section 3.3.4, in the event it 
proved quite difficult to make contact with such pupil users. Seventy pupils, who said on 
their questionnaire that they were happy to provide further information and gave an e- 
mail address, were contacted by e-mail. A total of five replies were received. Of these 
pupils, three were registered with NRICH and had used ‘one-to-one’ and perhaps some 
of the other NRICH bulletin board facilities. The other two replies were from pupils who 
were not registered and had never used any of the NRICH bulletin board facilities. 
Interview schedules for the e-mail conversations are given in Appendix E 

With this small response rate it was impossible to carry out any random sampling of 
pupils. All that was possible was to enough data to present three very brief accounts 
(given below), one a home user of primary age, one a secondary school pupil who 
accessed NRICH from the school IT suite, and one a student at a 16-19 college who 
accessed NRICH at home. The accounts cannot be taken as representative of pupils and 
students who access NRICH. 

7.2 A Home User of Primary Age 

This pupil was 10 years old and attended a private preparatory school. The pupil 
accessed the NRICH site occasionally (mostly less than once a month) from home. The 
bulletin board facilities were also accessed less than once a month. The pupil found out 
about NRICH from a parent and was not recommended to try NRICH by a teacher. The 
pupil found NRICH much better than the mathematics at school where mathematics was 
not “fun” and there was no mathematics club for interested pupils. The pupil thought that 
accessing NRICH had made mathematics seem more interesting. The pupil was more 
likely to want to continue studying mathematics as a result of using NRICH. 

The pupil had used ‘one-to-one’ several times and had very much enjoyed the exchanges 
with the University student who answered the queries. In each case, the replies were 
helpful and led on to further exchanges. The pupil greatly valued the opportunity of 
being able to partake in such exchanges. Few of the exchanges related to the 
mathematical problems provided on the NRICH site. 



7.3 A Secondary School Pupil 

This student was 16 years old and attended a secondary comprehensive school. The 
student accessed NRICH from the IT suite at school less than once a moth. The bulletin 
board facilities were also accessed less than once a month. The student had not been 
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recommended by a teacher to try NRICH but had found it while browsing the web. The 
student was unsure whether or not NRICH had made mathematics more interesting but it 
had raised the profile of mathematics as a subject worth continuing to study. 

The student had used ‘one-to-one’ on a few occasions. The student’s queries were 
always related to the mathematics examination syllabus the student was following rather 
than anything to do with the problems appearing on the NRICH site. The replies 
invariably made the student want to ask more questions. The student was very 
appreciative of this NRICH facility. 



7.4 A Student at a 16-19 College 

This student was 17 years old and attended a tertiary college studying mathematics at 
advanced (University entrance) level. The student accessed NRICH at the college where 
online access was free for students. When the student had identified appealing 
mathematical problems, the NRICH site was accessed from home and print outs made of 
these problems (printing was cheaper from home than at the college). NRICH was used 
in this way by this student about once a month. The NRICH site was recommended to 
the student by a teacher but using NRICH had not made the student more interested in 
studying mathematics. 

The student enjoyed tackling the more challenging problems but only rarely accessed any 
other part of the NRICH site. The student had made use of the ‘one-to-one’ facility on 
one occasion to ask a mathematical question not related to any of the NRICH problems. 
The student received a reply which was found to be very informative and said exactly 
what the student wanted to know. More often the student used the other bulletin boards 
(such as the ‘open discussions’ and the ‘Statue of Anonymous’) to pose questions or join 
in a discussion as the student was interested in replies from other students in the same 
position. 



7.5 Summary 

It was only possible to collect data on pupil usage of NRICH for three pupils. All three 
accessed NRICH no more than once a month. All three had found the ‘one-to-one’ 
facility helpful and informative, and often found themselves wanting to know more. All 
three valued the opportunity of being able to ask questions, although few of the 
exchanges they had through ‘one-to-one’ were related to the mathematical problems 
provided on the NRICH site. 
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8. Conclusions 



8.1 Introduction 

The objectives of the external evaluation were to assess how the use of the NRICH 
website facilities enhances the mathematical development of children who have the 
potential to go on to study mathematical subjects at university, how the features of the 
website are used by teachers to help meet the special educational needs of exceptionally 
able children in mathematics, and the particular contribution of Information 
Communications Technology (ICT) to the above. These objectives were derived from 
the aims of the NRICH project. In this section, evidence from the evaluation is used to 
draw conclusions on each of the objectives of the evaluation. 

8.2 The Impact of NRICH on Pupils 

Evidence from this evaluation suggests that pupils using the NRICH website facilities 
gained by having access to interesting mathematical problems. For some pupils, these 
mathematical problems were more stimulating than the mathematics they regularly did at 
school. Many pupils who accessed NRICH did so from home which is an indication that 
the NRICH materials are intriguing enough to attract pupils in their own time. Some 
pupils accessed NRICH quite frequently, another indication of the quality of the 
materials. Only a minority of pupils made use of the bulletin board facilities available 
through NRICH. Those that did so spoke highly of the service and how it stimulated 
further thought. These pupils particularly valued the opportunity of being able to ask 
mathematical questions and receive replies. Seeing their solutions published on the 
NRICH website was also popular with pupils. 

Girls were under-represented as NRICH pupil users. Certain ethnic groups might also 
have been under-represented but the numbers of respondents was not sufficient to draw 
any firm conclusions. Data on the socio-economic class of pupils was not collected (for 
reasons outlined in section 3.3.1) but the large proportion of pupils who accessed 
NRICH at home is one indication of the socio-economic status of their families. Few 
pupils accessed NRICH through a public library or other public access location. The 
main impact of NRICH on the more able pupils was in terms of helping them to gain a 
wider appreciation of mathematics and raising the profile of mathematics as a subject that 
could be interesting enough to pursue either within or outside school or for further study. 
Quantifying this impact was beyond the scope of this evaluation. 

8.3 Teachers’ use of NRICH 

Teachers mostly accessed NRICH to find problems to use in their teaching. Most 
worked in the state sector, began using NRICH relatively recently, accessed NRICH 
from their home, and were not registered (and thus did not have access to the NRICH 
bulletin boards). These teachers did recommend the site to their pupils and thought the 
site particularly good for those pupils who had a talent in mathematics. The teachers 
used a variety of approaches to meet the needs of their more able pupils. Some used the 
NRICH problems with groups of more able children withdrawn from their regular 
classrooms. Some teachers used NRICH problems as extension material once regular 
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classwork was complete. For some, NRICH was one resource amongst many. Only a 
few organised an extra-curricular mathematics club based solely around NRICH. 

8.4 The Contribution of ICT 

The contribution of ICT to both the enhancement of pupils’ mathematical development 
and to how teachers made use of the NRICH facilities was associated with the 
functionality and accessibility of the NRICH site. The NRICH website was judged by the 
evaluation team to score highly in terms of aesthetic appeal, functionality, ease of 
navigation, and the quality of the content. NRICH compared very favourably with other 
sites that provided mathematical puzzles, games and problems, and/or an answering 
service. All categories of respondent to the questionnaires also complimented the 
NRICH project for the design and ease-of-use of the NRICH website. Teachers 
preferred NRICH as a web-based service rather than a printed resource. Being on the 
world wide web meant that NRICH was accessible (providing that internet access was 
available and the NRICH server was reliable) and, unlike paper, it does not get lost under 
other paperwork. The site can also give pupils the impression that they are using a piece 
of software, which, in some schools, was important. 

Although most teachers did not use the bulletin boards nor send in pupil solutions, those 
that did valued these facilities very highly. The interaction that was possible through 
using information and communications technology was seen as a particular advantage of 
the NRICH project. 

8.5 Unexpected Outcomes 

One unexpected outcome of the evaluation was the limited number of completed 
questionnaires from registered NRICH users. The evidence from the evaluation indicates 
that there were some NRICH users who make frequent use of the NRICH website and 
some use of the NRICH bulletin boards. The majority of users, however, were relatively 
new to NRICH and were yet to make use of the wider NRICH facilities (such as the 
bulletin boards or sending in pupil solutions). It is unclear how many registered users 
were active users of NRICH. 

The evaluation also suggests that NRICH may be having a wider impact than might be 
expected giving its mission. The evidence from teachers suggests that many used NRICH 
as a source of mathematical problems in their teaching rather than a resource solely or 
mainly for their most able pupils. 

8.6 Suggestions for the Future Development of the NRICH Project 

The evidence from the evaluation allows the evaluation team to highlight some issues 
that may be worthy of consideration by the NRICH project in order to inform its future 
development. The following suggestions are not in any order of importance or priority: 

1. While the design of the NRICH site launched in July 1999 is a great improvement on 
the previous design, there remain differences between the ‘NRICH primary site’ and 
the ‘ main ’ site. For example, the primary ‘site’ has a ‘staffroom’ section and ‘teachers 
notes’ to accompany the mathematical problems, neither of which appear to be 
aspects of the ‘main’ site. There are also minor differences in terminology such as 
‘library’ on the primary ‘site’ and ‘archive on the ‘main’ site. Neither ‘site’ appears to 
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make any mention of parents. Some reconsideration of the intended audience for the 
various components of NRICH and the relationship between the components would 
aid the clarity of the NRICH website and should help new users to make best use of 
the material. 

2. The NRICH registration process would benefit from some attention. While registering 
is relatively straightforward, there appears to be no clear way of how to change 
registration details (for example, when moving schools, or changing postal address or 
e-mail account), nor how to de-register. There is also no mention during the 
registration process of the (UK) Data Protection Act, how the data will be used (for 
example, that it will not used for unsolicited mailings, nor sold to a third party), or the 
security of registration process. 

3 . Pupils said that they liked seeing their solutions published on the NRICH site. 

Teachers asked for a simpler way of submitting solutions. The stance that NRICH 
takes to pupil solutions would benefit from a review. The current design of the site 
does not make it clear whether pupil solutions are welcomed nor how such solutions 
can be submitted. Some information is present but, at the time of writing, the details 
are not that easy to find. 

4. Most pupil users of NRICH were boys. There was some evidence that particular 
ethnic groups might be under-represented as NRICH pupil users (although the 
numbers were too small to be at all certain about this). A large proportion of NRICH 
users accessed the site from home while access from public libraries was very low. 

The NRICH project might like to consider how it could reach those categories of 
users currently under-represented. 

5. Usage of the NRICH site is likely to continue increasing. All categories of current 
user valued the NRICH website very highly. As internet access and levels of teacher 
competency grow, both word-of-mouth and links to NRICH from sites such as the 
UK Virtual Teacher Centre 14 are likely to increase the number of NRICH users. This 
means that it will become increasingly important to be able to predict the likely impact 
on various NRICH resources. These include technical issues such as the bandwidth 
and reliability of the NRICH server, but also human resources such as those required 
to answer queries, monitor the bulletin boards, review and select pupil solutions, as 
well as continue the NRICH publishing schedule. 

6. Attention might be paid to the NRICH mission and to the aims, objectives, and 
method of the project now that it is entering a phase of further development. The aim 
of such a review would be to ensure a close match between the mission of the project, 
the facilities provided through the NRICH website, and the allocation of human and 
physical resources. Where possible, it would be helpful if objectives could be specified 
in a way that might make it possible to quantify outcomes. 

7. The review of physical resources might include a consideration of recently developed, 
and relatively sophisticated, web server statistics software and web-based 
questionnaire software in order for the NRICH project to generate for itself 
quantifiable data to inform the ongoing development of the project. 

8. A number of users made suggestions that the NRICH project might usefully consider. 
Amongst these suggestions were making the site more interactive so that some of the 



' 4 The UK Virtual Teacher Centre can be found at http://vtc.ngfl.gov.uk/ 
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mathematical problems might be solvable on screen, providing a (compressed) version 
of each monthly edition so that these could be downloaded and viewed offline, and 
circulating registered members with a monthly e-mail giving the highlights for each 
edition. 

8.7 Closing Comments 

The NRICH site was highly valued by its users. One user commented that since internet 
resources are notorious for being present one day and gone the next, it was hoped that 
NRICH would be around long enough for users to make full use of the resource. It is 
likely that many current users, as well as potential future users, would agree with this 
sentiment. 
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Appendix A: about the NRICH online mathematics project 

The following notes are taken verbatim or near verbatim from the general description of 
the NRICH project provided on the NRICH site and the details of its aims, objectives 
and methods (see web pages http://nrich.maths.org.uk/mathsfrabout.htm and 
http://nrich. maths, org. uk/mathsfraims . htm) . 

NRICH is located at the University of Cambridge School of Education and is a 
partnership between the University of Cambridge Faculty of Education and the Royal 
Institution. It began as a research and development project and has become a component 
of the Millennium Mathematics Project (MMP) a new national initiative, based in 
Cambridge, UK. The URL for the NRICH website is http:Wnrich.maths.org.uk 

The NRICH Mission Statement 

The project aims to establish a permanent national centre for curriculum enrichment to 
provide mathematical learning support for very able children of all ages. The learning and 
enjoyment of mathematics will be promoted through an Internet Newsletter and the 
participation of university students as peer teachers providing an electronic answering 
service. The centre will offer support, advice and inservice training to teachers, and 
resources for mathematics clubs. 

The aims of NRICH 

1 . To pave the way for the establishment of a permanent national UK Mathematics 
Enrichment Centre. 

2. To raise the standards of achievement in school mathematics, to promote the 
mathematical development of children who have the potential to go on to study 
mathematical subjects at university, and to support the special educational needs of 
exceptionally able children. 

3. To extend the provision of the Royal Institution Mathematics Masterclasses by 
providing continuous and sustained support for children so that they can participate 
wherever they live or go to school as individuals or as members of a school 
mathematics club. 

4. To develop the use of Information Communication Technology to provide interactive 
links to the centre and to facilitate links between schools and also between individual 
children. 

5. To extend peer assisted learning into a distance learning mode and so to contribute to 
the personal and cognitive development of both the pupils and the peer teachers. 

6. To promote and support the setting up of locally organised user groups and 
mathematics clubs by providing resources on the Internet, and offering advice and 
inservice training for teachers. 

7. To conduct research into the effect of communication technology and peer assistance 
on the quality of learning for very able pupils, on the quality of teaching offered by 
schools, on the cognitive gains for peer teachers and learners, and on the development 
of increased usage of IT in mathematics teaching. 
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Method 

1. The project will advance the development of mathematical thinking and language, and 
emphasise the importance of proof, through Internet Newsletters providing a regular 
fresh supply of mathematical challenges and problems, together with solutions 
contributed by the children. 

2. The NRICH Maths Centre will provide an electronic answering service whereby 
young people will be able to ask mathematical questions which will be answered 
personally by students from the University of Cambridge. Quality control will be 
exercised through rigorous selection and training of volunteers, and through checking 
samples of question and answer exchanges. 

3. All services of the NRICH project are freely available to all schools throughout the 
UK 15 . 



i 



15 In fact, registration with NRICH is free to schools anywhere and to pupils and others at home or 
wherever. 
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Appendix B: website impact questionnaires 



This appendix contains copies of the three questionnaires mounted on the NRICH 
website during May 1999 (from 1 May 1999 to 6 June 1999). Being web-based 
questionnaires, for many of the questions respondents could choose their responses from 
drop-down menus or ‘click’ a mouse button to make their choice. Questionnaire 
responses were stored electronically and the results converted to computer files suitable 
for analysis using a spreadsheet. 

The three questionnaires are aimed at the following groups and appear in this appendix in 
that order: 

• school or college teachers 

• school pupils or students 

• those who categorise themselves as neither a school or college teacher nor a school 
pupil or student 



Evaluation of NRICH 

NRICH has asked the University of Southampton to conduct an independent evaluation 
of the NRICH online mathematics project. The aim is to provide information to NRICH 
in order to further develop the project. 

Your help is much appreciated in spending a few minutes completing this questionnaire. 

All individual responses will be kept strictly confidential. 

For most of the questions you choose from a drop-down menu. 

To change an answer, click on another choice 

Please choose one of the following: 

• I am a school or college teacher 

• I am a school pupil or student 

• I am neither a school teacher nor a school pupil or student 



Notes 

1 . On the web-based versions of these questionnaires the options above were hypertext 
buttons that led to the different versions of the questionnaire. 

2. The web-based questionnaires that follow used drop-down menus for questions 1-9. 
The default choice was “choose” which was recorded as a nil-response. 
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Teacher questionnaire (school or sixth-form teachers only please) 

For many of the questions you choose from a drop-down menu. 

To change an answer, click on another choice 

First some questions about your school or college 

1. Country 

• England 

• Northern Ireland 

• Scotland 

• Wales 

• Another country - please specifyi 



2. Type of School 

• primary (infant or junior) 

• middle 

• secondary comprehensive 

• secondary selective 

• 16-19 college 

• private (preparatory) 

• private (upper) 

• other - please specify 



3. School status (only for UK schools please ) 

• LEA 

• grant maintained 

• voluntary aided or controlled 

• further education 

• private 

• other - please specify 

4. School location 

• inner-city 

• urban 

• suburban 

• rural 

5. Number of pupils 

• less than 200 

• 200-499 

• 500-999 

• 1000 or more 



0 
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6. Pupils or students 

• boys 

• girls 

• mixed (all years) 

• mixed (some yeargroups) - please specify 



Now some questions about your own use of NRICH 

7. How did you first learn about NRICH? 

• a colleague 

• a leaflet 

• an article in a journal 

• an INSET course 

• from browsing the web 

• by another means- please specify 



8. How long have you been using NRICH? 

• a month 

• six months 

• a year 

• a year or two 

• more than two years 

9. Where do you usually access the NRICH site from? 

• school library 

• school staffroom 

• departmental office/room 

• own classroom 

• IT suite 

• home 

• elsewhere - please specify I 



If you access NRICH from more than one place, please explain 
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10. How often do you use the following NRICH facilities 





most 

days 


most 

weeks 


once a 
month 


less 
than 
once a 
month 


never 


Ask NRICH (answering your mathematical queries) 


0 


0 


0 


0 


0 


Emailing list or OSkNRICH (to follow mathematical 
discussions) 


0 


0 


0 


0 


0 


Primary Site: Bernard's Bag 


0 


0 


0 


0 


0 


Penta Problems 


0 


0 


0 


0 


0 


Let Me Try 


0 


0 


0 


0 


0 


Kids Mag 


0 


0 


0 


0 


0 


Staff Room 


0 


0 


0 


0 


0 


Main Site: News 


0 


0 


0 


0 


0 


Articles 


0 


0 


0 


0 


0 


Monthly Six Problems 


0 


0 


0 


0 


0 


More Challenging Problems 


0 


0 


0 


0 


0 


Play Games (Primary) or Games (Secondary) 


0 


0 


0 


0 


0 


Resource Bank/Library 

(stored problems etc. search, and links to other sites) 


0 


0 


0 


0 


0 


Send in your pupils' solutions 


0 


0 


0 


0 


0 



1 1 . Please say how far you agree or disagree with each of the following statements 





Strongly 

agree 


Agree 


Disagree 


Strongly 

disagree 


don’t know 


The NRICH website is well-designed 


0 


0 


0 


0 


0 


It is difficult to find what you are 
looking for on the NRICH website 


0 


0 


0 


0 


0 


I don’t have the time to use the ask 
NRICH bulletin board 


0 


0 


0 


0 


0 


One-to-One (ask a mathematician) is 
the best facility provided by NRICH 


0 


0 


0 


0 


0 


NRICH makes me feel I can share 
issues with other teachers 


0 


0 


0 


0 


0 


I always look forward to new NRICH 
monthly editions 


0 


0 


0 


0 


0 


I never read the articles on the main 
site 


0 


0 


0 


0 


0 


The problems are the best part 


0 


0 


0 


0 


0 


Sending in solutions is too time 
consuming 


0 


0 


0 


0 


0 


The Resource Bank/ Library is the only 
part of NRICH I regularly use 


0 


0 


0 


0 


0 
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12. Please say how far you agree or disagree with each of these statements 





Strongly 

agree 


Agree 


Disagree 


Strongly 

disagree 


don’t 

know 


I mainly use NRICH as a source of 
problems to use in my teaching with my 
classes 


0 


0 


0 


0 


0 


I never suggest to any of my pupils that they 
access NRICH independently 


0 


0 


0 


0 


0 


My school has a maths club based around 
the NRICH facilities 


0 


0 


0 


0 


0 


I only recommend NRICH to the most able 
pupils in my classes 


0 


0 


0 


0 


0 


I mostly use NRICH for setting homework 


0 


0 


0 


0 


0 


My pupils will never have heard of NRICH 


0 


0 


0 


0 


0 


NRICH is particularly good for those pupils 
of mine that have a talent for mathematics 


0 


0 


0 


0 


0 


There is no advantage for NRICH to on the 
www. It would be better as a printed 
magazine. 


0 


0 


0 


0 


0 


NRICH is just an entertaining pastime, it is 
not an important way for me to develop my 
pupils’ mathematics 


0 


0 


0 


0 


0 


Using NRICH with my pupils has made 
them more interested in mathematics 


0 


0 


0 


0 


0 



Is there anything else you wish to add which indicates how you use NRICH and what 
value you think it has? 



And finally, please indicate if you are 

♦ registered with NRICH. My NRICH ID, if known, is 

l ~ 

(this information will only be used to count responses from NRICH members) 

♦ happy to be contacted by e-mail to provide further information 
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My e-mail address is I 



Before clicking on Questionnaire complete, please check that you have not accidentally 
missed a question. 

Thank you for your help. 

A summary of the evaluation will be published on the NRICH website. 

Questionnaire complete 
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Pupil or student questionnaire 

For many of the questions you choose from a drop-down menu. 
To change an answer, click on another choice 

First some questions about where you go to school or college 

1 . I live in 

• England 

• Northern Ireland 

• Scotland 

• Wales 

• Another country - please specify 




2. My school is 

• primary (infant or junior) 

• middle 

• comprehensive 

• grammar 

• 16-19 college 

• private (preparatory) 

• private (upper) 

• other - please describe 



3. My school or college is 

• in a large city 

• in a small town 

• in the countryside 

4. The pupils or students in my school or college are 

• boys/male 

• girls/female 

• both male and female 

Now some questions about your use of NRICH 

5 . I usually use NRICH when I am 

• in the school library 

• in my mathematics classroom 

• in another classroom (not mathematics) 

• in the IT room 

• at home 

• in the public library 

• elsewhere - please say where 
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If you access NRICH from more than one place, please explain 



6, How often do you use the following parts of NRICH? 





most 

days 


most 

weeks 


once a 
month 


less than 
once a 
month 


never 


Ask NRICH (answering your mathematical queries) 


0 


0 


0 


0 


7 0 


Emailing list or askNRICH (to follow mathematical 
discussions) 


0 


0 


0 


0 


0 


Primary Site: Bernard's Bag 


0 


0 


0 


0 


0 


Penta Problems 


0 


0 


0 


0 


0 


Let Me Try 


0 


0 


0 


0 


0 


Kids Mag 


0 


0 


0 


0 


0 


Staff Room 


0 


0 


0 


0 


0 


Main Site: News 


0 


0 


0 


0 


0 


Articles 


0 


0 


0 


0 


0 


Monthly Six Problems 


0 


0 


0 


0 


0 


More Challenging Problems 


0 


0 


0 


o 


0 


Play Games (Primary) or Games (Secondary) 


0 


0 


0 


0 


0 


Resource Bank/Library 

(stored problems etc. search, and links to other sites) 


0 


0 


0 


0 


0 


Send in my own solutions with the help of a teacher 


0 


0 


0 


0 


0 


Send in my solutions on my own 


0 


0 


0 


0 


0 



7. Please say how far you agree or disagree with each of the following statements 





Strongly 

agree 


Agree 


Neither 


Disagree 


Strongly 

disagree 


don’t 

know 


The NRICH website is well- 
designed 


0 


0 


0 


0 


0 


0 


It is difficult to find what you are 
looking for on the NRICH website 


0 


0 


0 


0 


0 


0 


The ask NRICH bulletin board is 
difficult to use 


0 


0 


0 


0 


0 


0 


One-to-One (ask a mathematician) 
is the best facility provided by 
NRICH 


0 


0 


0 


0 


0 


0 


NRICH makes me feel part of a 
mathematics club 


0 


0 


0 


0 


0 


0 


I always look forward to new 
NRICH monthly editions 


0 


0 


0 


0 


0 


0 


I never read the articles 


0 


0 


0 


0 


0 


0 


The problems are the best part 


0 


0 


0 


0 


0 


0 


I don’t understand why games are 
included. Games are not 
mathematics. 


0 


0 


0 


0 


0 


0 


I like seeing my solutions published 
on the world wide web 


0 


0 


0 


0 


0 


0 
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8. Please say how far you agree or disagree with each of these statements 





Strongly 

agree 


Agree 


Neither 


Disagree 


Strongly 

disagree 


don’t 

know 


My teacher suggested that I try 
NRICH 


0 


0 


0 


0 


0 


0 


NRICH is better than the 
mathematics I do at school 


0 


0 


0 


0 


0 


0 


My school has a maths club where 
we always use NRICH 


0 


0 


0 


0 


0 


0 


I usually find mathematics easy 


0 


0 


0 


0 


0 


0 


I never use NRICH in school 


0 


0 


0 


0 


0 


0 


I always work with a friend on 
NRICH problems 


0 


o 


0 


0 


0 


0 


I find all the NRICH problems too 
hard 


0 


0 


0 


0 


0 


0 


Using NRICH has made me more 
interested in mathematics 


0 


0 


0 


0 


0 


0 


My friends think I am mad to like 
NRICH 


0 


0 


0 


0 


0 


0 


Using NRICH has made me think 
that I would like to continue 
studying mathematics 


0 


0 


0 


0 


0 


0 


I would prefer NRICH to be a 
printed magazine 


0 


0 


0 


0 


0 


0 



Is there anything else you wish to add which tells us how you use NRICH and what you 
think of it? 




Now some questions about you. 
9. My age is: I 



10 I am a 

• boy/male 

• girl/female 
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11. I am 

• White 

• Black - Caribbean 

• Black - African 

• Black - Other 

• Indian 

• Pakistani 

• Bangladesh 

• Chinese 

• Other Asian groups 

• Other groups 

And finally, please indicate if you are 

• registered with NRICH. My NRICH ID, if known, is 



(this information will only be used to count responses from NRICH members) 
♦ happy to be contacted by e-mail to provide further information 
My e-mail address is I 



Before clicking on Questionnaire complete, please check that you have not accidentally 
missed a question. 

Thank you for your help. 

A summary of the evaluation will be published on the NRICH website. 

Questionnaire complete 
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Questionnaire for friends of NRICH 
For many of the questions you choose from a drop-down menu. 
To change an answer, click on another choice 

First some questions about you 

1. I am 

• a parent 

• a lecturer in higher education 

• a local education authority advisor 

• an inspector 

• a school governor 

• an adult studying mathematics 

• an adult interested in mathematics 

• a student teacher 

2. I live or work in 

• England 

• Northern Ireland 

• Scotland 

• Wales 

• Another country - please specify] 



3. I heard about NRICH from: 

• a colleague 

• another family member 

• a leaflet 

• an article in a journal 

• an INSET course 

• from browsing the web 

• by another means- please specify 



4. I have been accessing NRICH for 

• a month 

• six months 

• a year 

• a year or two 

• more than two years 

5. I usually access NRICH from 

• work 
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• home 

• elsewhere - please specify 



Please indicate how you use NRICH 




Please tell us what you like about MUCH 



Please tell us what could be improved 



And finally, please indicate if you are 

♦ registered with MUCH. My NRICH ID, if known, is 

I 1 

(this information will only be used to count responses from NRICH members) 

♦ happy to be contacted by e-mail to provide further information 
My e-mail address is I 

erJc 



86 



94 



Online Mathematics Enrichment 



Before clicking on Questionnaire complete, 
please check that you have not accidentally missed a question. 

Thank you for your help. 

A summary of the evaluation will be published on the NRICH website. 

Questionnaire complete 
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Appendix C: website evaluation criteria 

The website evaluation criteria below were adapted from those described by Branch et al 
(1999), Coe and Land (1998), Shneiderman (1997), and by Testa (1998) and cover the 
following: 

• Authority 

• Accuracy 

• Currency 

• Navigation and Design 

• Applicability and Content 

• Scope 

• Audience Level 

• Quality 

The categories are not entirely discrete; there is inevitably some overlap. 

1. AUTHORITY 

• Is the information credible and of high quality? 

• Is the information objective? 

• Is the author clearly identified? 

• What are the author's professional affiliations? 

• Can the author be contacted? 

• Is this the official site of an organization or association? 

2. ACCURACY 

• Is the information correct? 

• Does any advertising create a possible conflict of interest? 

3. CURRENCY 

• How frequently is the site updated with new information? 

• Are different parts of web sites updated at different times? 

• Is the date of updating clearly visible? 

4. NAVIGATION AND DESIGN 

• Does the site function well and allow the user to progress logically through the 
information? 

• Does the arrangement of elements when viewed in a browser window serve to 
make the purpose of the site clear? 

• Are there enough internal links? 

• Are the links to other resources kept up to date? 

• Are all the links relevant to the nature of the site? 

• Is the content presented in an orderly manner? 

• Does the design promote understanding of the content? 

• Is a text-only version available? 

• Is the overall design ‘user friendly’? 

• Is the overall design of the site aesthetically pleasing? 
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• Is there a search facility? 

5. APPLICABILITY AND CONTENT 

• Is the content relevant? 

• Is the content coverage sufficient? 

• Is the purpose of the content clear? 

• Is there variation in how the content is presented? 

6. SCOPE 

• What is included? 

• What is the purpose of what is included? 

• Does the site match its own mission or scope statement? 

7. AUDIENCE LEVEL 

• Is the intended audience clear? 

• Is all the material presented consistent with the intended audience? 

• Is the site slow to load? 

• Are all parts of the site easily accessible? 

8. QUALITY OF WRITING 

• Does all the text follow the essential conventions of spelling and grammar? 

• Is the use of unnecessary jargon kept to a minimal? 

• Are complex ideas introduced and discussed with clarity? 

• Is humour used effectively? 

9. REVIEWS 

• Has the site won any awards? 



erIc 
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Appendix D: case study interview foci 

During the preparation of the case studies, the following foci were used to inform the 

interviews and classroom observations, and the e-mail conversations: 

1 . General information on the school and its pupils. 

2. The approach to mathematics teaching. 

3. The use of ICT in mathematics teaching 

4. Details of examination entries and number of students going on to study mathematical 
subjects at university (if appropriate). 

5. Details of any provision for exceptionally able students in mathematics 

6. Use of NRICH 

7. Evidence of possible impact of NRICH on mathematics teaching and learning 

As far as was possible, school visits included the following: 

1. Time to talk with the teacher contact (the teacher who completed the NRICH teacher 
questionnaire). 

2. Time to observe some mathematics lessons to gain insight into the approach to the 
teaching of mathematics used in the school. 

3 . Time to see the ICT resources. 

4. If appropriate, time to interview some pupils (with the contact teacher, or another 
teacher, present) who have accessed NRICH themselves or made use of any NRICH 
facilities. 
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Appendix E: case profile interview schedule 

Interviews with the case profile pupils were planned as conversations to be conducted by 
e-mail. The schedule for the interviews were planned to be conducted along the 
following lines: 

1. Use of the ‘One-to-One’ facility on the NRICH site where help with a mathematical 
problem can be obtained: 

has the pupil used ‘One-to-One’? 

if yes, when was last time? 

what was the nature of your problem? 

what sort of reply was received and how long did it take to get a reply? 
was the reply helpful? 

has ‘One-to-One’ been used more than once? 
if so, how often? 

2. Use of any other NRICH bulletin Board facility? 

what other NRICH facilities have been used? 
for what purpose? 

how are the other facilities different to ‘One-to-One’? 
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University of Southampton Centre for Research in Mathematics Education 

The Centre for Research in Mathematics Education (CRiME) is a research centre of the 
University of Southampton, UK. Active members of the centre come from the Research and 
Graduate School of Education and the Faculty of Mathematical Studies. 

Work in mathematics education at the University of Southampton has had a national and 
international reputation for almost forty years. Activities of the centre include research and 
curriculum development in mathematics education and professional development opportunities 
for teachers and other education professionals. The Centre has close links with the School 
Mathematics Project, which began as a University research project in 1963. 

For more information on the work of the centre, please contact Professor Gary Davis, Director, 
Centre for Research in Mathematics Education, Research and Graduate School of Education, 
University of Southampton, Highfield, Southampton SOI 7 1BJ, United Kingdom, or visit the 
centre’s website: 

http://www.soton.ac.uk/~gary/crime.html 
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